:W&Bá¾»úÒM wn0bavzoª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0~Bù‚" n1papkc8443z wn0bavzoŠÕ wn0bavzojuju11-uc-san-diegoSPAtop"™ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.25922336132004453 epochs‚10  hidden_dim‚128 & learning_rate‚0.001685492695283354  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚10  num_encoder_layers‚6Bjjtttt_blocks_wn0bavzobs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pŠ ¤ÃëÔ͈5š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64g¬7 wn0bavzo&  _wandb.runtime‚0  _runtime‚0¦ÕDjÒä -Linux-6.8.0-138-generic-x86_64-with-glibc2.36CPython 3.12.10 £ÃëÔ°Í»Ü*-of*=/data/spatop/options_files/sweep_tttt_jj/config_wn0bavzo.json*-l* /scratch/logs*-n*jjtttt_blocks_wn0bavzo2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pr/opt/conda/bin/pythonº 9t2ltnl57x1zo4u5qhebb2030yng03o2óôPÖÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€àÃŽç$ª€ äÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-8cb16037-0025-0eda-9425-82f6843a99c3Ê13.2º 9t2ltnl57x1zo4u5qhebb2030yng03o2µª½Ç2  requirements.txtÝŒ8W wn0bavzoZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64ongÆY wn0bavzoZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64CÍl*  wn0bavzob trainer/global_step2óÓè\  wn0bavzoZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64¯,¬/  wn0bavzob*"trainer/global_step2r¾SˆK  wn0bavzoj: ¤ÃëÔèÌÑþ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] lV"’  wn0bavzoj€  ¥ÃëÔ€ÞÀyî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 866 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 2.0 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 53.9 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ L®[ wn0bavzojÉ ¥ÃëÔÈþz·Trainable params: 56.7 M Non-trainable params: 86 Total params: 56.7 M Total estimated model params size (MB): 226.908 Modules in train mode: 5928 Modules in eval mode: 0 Total FLOPs: 0 Dp wn0bavzoj_ ¥ÃëÔ ¦ÚzPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ³å'z wn0bavzoji ¥ÃëÔàÌâzZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ©¨Ãúx wn0bavzojg ¥ÃëÔ¨‘îzXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ~Hÿ wn0bavzojn ¥ÃëÔ€­÷z_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). ¥*Ðy wn0bavzojh ¥ÃëÔÀÖþzYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update).  U¯s wn0bavzojb ¥ÃëÔÀÄ„{Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). # ‚p wn0bavzoj_ ¥ÃëÔŠ{Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). \Ó Øs wn0bavzojb ¥ÃëÔø·{Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). þ‡¢m wn0bavzoj\ ¥ÃëÔ؃—{Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ¿.Ôñl wn0bavzoj[ ¥ÃëÔè¿{Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ÃÆÜ wn0bavzo*Ê  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"D{Êë wn0bavzojÙ ¥ÃëÔˆ¡óÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ù°ä&N wn0bavzoj¼ ¥ÃëÔè΃‚«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. #žtéë wn0bavzojÙ ªÃëÔ¨ÏÅâÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ¤}uP wn0bavzoj¾ ªÃëÔ ÍŒã­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ¹Úæ#’‚(:ˆ ³ÃëÔÀ¨•Ð gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.404861450195312* gpu.0.memoryAllocatedBytes‚ 14950662144 gpu.0.temp‚30.0 gpu.0.powerWatts‚116.732' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šm1Ò;‚(:± ³ÃëÔºïÛ disk.dm-0.in‚ 0.0078125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0 network.recv‚69757 disk.nvme0n1.in‚ 0.0078125 disk.nvme0n1.out‚ 235.140625 disk.sda3.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 0.0078125! disk.nvme0c0n1.out‚ 235.140625' proc.memory.availableMB‚ 963.2421875$ disk./.usageGB‚1177.754825592041 disk.sda.out‚0.453125 disk.dm-1.out‚0.453125 disk.sda2.out‚0$ memory_percent‚94.12083625793457# proc.memory.rssMB‚ 2954.99609375) proc.memory.percent‚18.03586483001709 cpu‚26.869143051206677 network.sent‚48708 disk.sda3.out‚0.453125 disk.dm-0.out‚ 235.140625 proc.cpu.threads‚17( disk./.usagePercent‚79.0401145476015 disk.sda.in‚0 disk.sda1.out‚0¸^˜ ‚(:† ÂÃëÔ³›ìÏ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.735' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.947 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…æû]]!‚(:Ó ÂÃëÔ¡´‹Ô proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.00390625 disk.nvme0c0n1.in‚ 0.0078125 disk.dm-0.in‚ 0.0078125$ memory_percent‚94.13797855377197* proc.memory.percent‚18.043875694274902 network.recv‚74209 disk.dm-1.out‚ 1.13671875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚26.571297716671143 network.sent‚60402& disk.nvme0c0n1.out‚653.19775390625$ disk.nvme0n1.out‚653.19775390625 disk.sda3.out‚ 1.13671875 disk.sda.in‚0# proc.memory.rssMB‚ 2956.30859375) disk./.usagePercent‚79.04011582763918% disk./.usageGB‚1177.7548446655273 disk.nvme0n1.in‚ 0.0078125 disk.sda.out‚ 1.13671875! disk.dm-0.out‚653.19775390625( proc.memory.availableMB‚ 960.43359375(k¸b‘"‚(:‡ ÑÃëÔÈ“§Ù gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.629' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nžAa#‚(:× ÑÃëÔÿÞžæ) disk./.usagePercent‚79.04168950595215! disk.nvme0c0n1.in‚ 193.4921875 disk.sda.out‚ 1.75390625 disk.dm-0.in‚ 193.4921875 cpu‚26.36723964846027% disk./.usageGB‚1177.7782936096191 disk.sda2.in‚0 disk.drbd1260.out‚ 0.00390625 disk.sda3.out‚ 1.75390625 disk.sda.in‚0' disk.nvme0c0n1.out‚1060.65087890625% disk.nvme0n1.out‚1060.65087890625 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚94.14052963256836& proc.memory.availableMB‚ 960.015625 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 193.4921875" disk.dm-0.out‚1060.65087890625 disk.dm-1.out‚ 1.75390625! proc.memory.rssMB‚ 2956.359375* proc.memory.percent‚18.044185638427734 proc.cpu.threads‚17 network.sent‚65448 network.recv‚79384PÆ5þ$‚(:“ àÃëÔ±ùïÔ gpu.0.gpu‚18.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.707' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.941399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.›à`%‚(:Ö àÃëÔ둇ß disk.sda3.out‚ 2.33984375 disk.sda.out‚ 2.33984375% disk./.usageGB‚1177.7782936096191 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0078125& disk.nvme0c0n1.out‚1123.7900390625 disk.nvme0n1.in‚ 193.4921875 disk.sda3.in‚0) disk./.usagePercent‚79.04168950595215$ disk.nvme0n1.out‚1123.7900390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2956.37109375 cpu‚26.2185751970685 proc.cpu.threads‚17 network.sent‚84973! disk.nvme0c0n1.in‚ 193.4921875 disk.dm-0.in‚ 193.4921875! disk.dm-0.out‚1123.7900390625 disk.dm-1.in‚0 disk.dm-1.out‚ 2.33984375' proc.memory.availableMB‚ 955.1796875 network.recv‚109537 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚94.17004585266113* proc.memory.percent‚18.044257164001465”}&‚(:“ ïÃëÔ“»Ñ gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.674799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øà.Z'‚(:Ð ïÃëÔÉÔ÷× network.sent‚93467% disk./.usageGB‚1177.7781982421875 disk.nvme0n1.in‚ 193.4921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 2.59765625! disk.nvme0c0n1.in‚ 193.4921875& disk.nvme0c0n1.out‚1469.3916015625$ disk.nvme0n1.out‚1469.3916015625 disk.sda3.out‚ 2.59765625 disk.sda.in‚0 disk.sda.out‚ 2.59765625 disk.dm-1.in‚0 disk.sda2.out‚0$ memory_percent‚94.15102005004883& proc.memory.availableMB‚ 958.296875 network.recv‚114685 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0078125* proc.memory.percent‚18.044281005859375 proc.cpu.threads‚17( disk./.usagePercent‚79.0416831057638 disk.sda3.in‚0 disk.dm-0.in‚ 193.4921875! disk.dm-0.out‚1469.3916015625 proc.memory.rssMB‚2956.375 cpu‚26.10600025265125`F}¡(‚(:† þÃëÔœÄÖÌ gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚35.0 gpu.0.powerWatts‚172.84' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì ¸d)‚(:Ú þÃëÔŠù™Ð disk.sda3.out‚ 3.16015625 disk.sda.out‚ 3.16015625 disk.dm-0.in‚ 193.4921875* proc.memory.percent‚18.044471740722656 cpu‚26.018000996747258 network.sent‚99670% disk./.usageGB‚1177.7782173156738 disk.drbd1260.out‚ 0.01171875! disk.nvme0c0n1.in‚ 193.4921875 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚94.15285587310791 network.recv‚119097' disk.nvme0c0n1.out‚1618.32373046875 disk.nvme0n1.in‚ 193.4921875" disk.dm-0.out‚1618.32373046875 disk.dm-1.out‚ 3.16015625( proc.memory.availableMB‚ 957.99609375 proc.memory.rssMB‚ 2956.40625 proc.cpu.threads‚17) disk./.usagePercent‚79.04168438580147 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚1618.32373046875øŽÝ•’*‚(:ˆ ÄëÔíëÒÎ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.749' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7498 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ÿL`+‚(:Ö ÄëÔ³ØËÓ" proc.memory.rssMB‚ 2956.4140625 proc.cpu.threads‚17 network.sent‚105244) disk./.usagePercent‚79.04010814741315 disk.sda2.in‚0 disk.drbd1260.out‚ 0.01171875" disk.nvme0c0n1.in‚ 212.49609375# disk.nvme0n1.out‚1868.029296875 disk.sda3.out‚ 3.55859375 disk.sda.out‚ 3.55859375 disk.sda2.out‚0 disk.dm-0.in‚ 212.49609375 disk.dm-0.out‚1868.029296875 disk.dm-1.in‚0 disk.dm-1.out‚ 3.55859375$ memory_percent‚94.15149688720703% proc.memory.availableMB‚ 958.21875 cpu‚25.94824290181401% disk.nvme0c0n1.out‚1868.029296875 disk.nvme0n1.in‚ 212.49609375 disk.sda1.in‚0* proc.memory.percent‚18.044519424438477 network.recv‚123268% disk./.usageGB‚1177.7547302246094 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0=¹.53,b/ lr-AdamW/pg1"trainer/global_step2:JP–S™Ü3-b/ lr-AdamW/pg2"trainer/global_step2:JPvyV/.+ _wandbruntime‚106  _runtime‚106¢¶ÞÿÀ/» _runtime‚ 106.031777388 _step‚0 & lr-AdamW/pg1‚6.831194546640558e-05 & lr-AdamW/pg2‚6.831194546640558e-05 trainer/global_step‚49 $ _timestamp‚1.788535310720711e+09/Úú¬Â0½ & lr-AdamW/pg1‚6.831194546640558e-05 & lr-AdamW/pg2‚6.831194546640558e-05 trainer/global_step‚49 $ _timestamp‚1.788535310720711e+09 _runtime‚ 106.031777388 _step‚0æ•%üp1bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPûÞ­óo2bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP¥–CÜa3b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP;ûq4bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP‹<"j5bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPëäâx6bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP>ˆIt7bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP5r'{k8bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP zÓKi9be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPJþWr:bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP¥B9–_;b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JP)ÀUhbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPeH’o?bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPLà=#g@bc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPµ^šØrAbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPùmu­]BbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP‚Ç9úmCbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPmíˆgDbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP7;yqEbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP8Dd,nFbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPDýn«mGbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPJÝIvHbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPV^™×rIbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPÙ0"rJbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP6²“lKbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPZypØmLbi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP9=g–jMbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP`²øeNba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP*˜n€nObj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP¬gahPbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP–WÁroQbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPÐ6æÊiRbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPDÏ_gSbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPô3PjTbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP{»`K_Ub[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPÛŒÖsVbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP§ GzfWbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPrcXb_ DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP,%i„bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPXÇÇm…bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPà‹É›m†bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPò(é¸t‡bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPXßN‹qˆbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP¹¾+j‰be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP†´smŠbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPÆ»n‹bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP÷ÕPpjŒbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP̆Ügbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP`6ÏgŽbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPÖð áobj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP‘œ¿îkbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPtF¶mj‘be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPïÈŒl’bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP hÂ*p“bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPŒx§Fm”bh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPGv#i•bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP°²Šq–bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP„ë`©w—br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP WbJf˜ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP² =(_™bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPN<–@cšb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP¨äqmc›b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPÎÁ ×mœbh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPëAgìnbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP(‡ßLužbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP!ªÿyŸbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPüIÖil bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPKü(k¡bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPÞ5tšk¢bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPóGt£bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPÇ"ŽAq¤bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP“t¥bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPuuAÿf¦ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPqmd"f§ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP9²ßh¨bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP5U oX©bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JP2û‡¸mªbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP«â€l«bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP•˜‰0¬+ _wandbruntime‚106  _runtime‚106 ÐCÓ­›O P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4160513990656 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚899944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4029149496644 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56630104576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1018792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4532299600384 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_poolŸÐˆÎÎ#.freed‚ 59143594496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚610300 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1235123 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4104793090560 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2278763 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚290258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚49 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1018794 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4591419254784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1259969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1259969 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3972714343024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1235123 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2253915 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1018792 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 55720900096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4104793090560 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1149 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1018794 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4161441198080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 55720900096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4028224489828 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4161441198080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4532275660288 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 59379435008 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚900558 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 56418114260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4104811093504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4104811093504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _runtime‚ 106.032846433/Êi‚¥'®ŸO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4161441198080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4532275660288 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 59379435008 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚900558 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 56418114260 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4104811093504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4104811093504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _runtime‚ 106.032846433 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4160513990656 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚899944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4029149496644 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56630104576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1018792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4532299600384 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 59143594496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚610300 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1235123 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4104793090560 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2278763 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚290258 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚49 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1018794 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4591419254784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1259969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1259969 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3972714343024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1235123 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2253915 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1018792 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 55720900096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4104793090560 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1149 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1018794 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4161441198080 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 55720900096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4028224489828 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749ôBd"p¯bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPÀU˜Ff°ba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPl=öªq±bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP(5Ö|m²bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPÖÝ™1k³bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP7Ó'j´be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP‡pu^µbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP±HÛg¶bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP{,Gq·bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP׌e¸b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPØ Yh¹bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP"†Zwfºba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPý=ãr»bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP6G˜Òk¼bf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPðC†Íq½bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPLù/˜a¾b\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JP‰ËÑÜg¿bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPNC@hÀbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JP¿]mÁbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPÂr ,`Âb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JP¨xr¡fÃba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPTO…çhÄbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPøµÔrÅbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP|dbôaÆb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP©bÈkgÇbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP HõeÈb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP/úÎQhÉbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPV= iÊbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPN«KcËb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPR^DmÌbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPË{hÍbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPýTÛ2oÎbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP¡3žÝoÏbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPØÎcïkÐbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP þ©ÉnÑbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP åÒfwÒbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP|±‡ÛuÓbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPŽa±/]ÔbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP7‘5FlÕbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPHMÃmÖbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPôè·j×be BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPAð=gØbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPáÀ^VÙbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP©%æ¥pÚbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPHjd‚eÛb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPhÅ¢ËkÜbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP¾ü9jÝbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPÄ*³ÊpÞbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPOüöGmßbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP{Jsoàbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP1Y¡Ãiábd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPðvPúcâb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP=G“…oãbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPa-E¬`äb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPIÒttjåbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPuÿyÌdæb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP䬵 ]êbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP<²Ãmëbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP/‰³nìbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP£§À líbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPá\îbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPï\BÒrïbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPJ¦_ðbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JPvÚâjñbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPJhºmòbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP>€jilóbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPSÃüpôbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPW¥^õbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPK‚ßgöbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPGm÷bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP€Lrøbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÆ»Ôkùbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP1¹†húbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPÚ‡*eûb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP¯œ"Žiübd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPrÌòˆiýbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP3¨Òdþb_ If‚ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPïBØrƒbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP³ÁWf„ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPO€p…bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPùv”Âq†bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP‰‹|?u‡bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPzSó,^ˆbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP,ea„m‰bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPšº‹°eŠb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPšßµ?d‹b_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPV^m bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPYÍw–m¡bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPÀ?‹o¢bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP–Š†Àn£bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP u¡b¤b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP¦~*´b¥b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JP@µt:¦bn KDeviceStatsMonitor.on_train_batch_end/inactive_spliÍ[”ë9t_bytes.small_pool.freed"trainer/global_step2:JPáA;ln§bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPòuÔµs¨bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP¡-±èf©ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP?’¥¥0ª+ _wandbruntime‚106  _runtime‚106&øn«&«¥M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 4051731937568 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2323780 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2298906 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 60313279488 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 56842360320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4622501364736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚917749 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚918362 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2298906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚622365 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4186430179328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1284826 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2323780 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57751578112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4108359357650 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4244181757440 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1038954 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 _step‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1038954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4186412176384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4186412176384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4622525304832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885353116525116e+09 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚295386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 56842360320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 57751578112 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1284826 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4186430179328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4682814644224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57535387746 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 106.964224178 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚622363 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1149 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4683068120064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4109284364674 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1259954 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚295997 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1186 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0‡—>-¯&¬©M WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 57751578112 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1284826 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4186430179328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4682814644224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57535387746 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 106.964224178 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚622363 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1149 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4683068120064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4109284364674 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1259954 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚295997 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1186 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 4051731937568 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2323780 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2298906 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 60313279488 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 56842360320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4622501364736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚917749 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚918362 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2298906 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚622365 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4186430179328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1284826 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2323780 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57751578112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4108359357650 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4244181757440 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1038954 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 _step‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1038954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 4186412176384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4186412176384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4622525304832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 % _timestamp‚1.7885353116525116e+09 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚295386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 56842360320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845šˆC­b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP¥QöB®b= loss/SRqqt3/detection_loss"trainer/global_step2:JP°£²@¯b; loss/FBt2/detection_loss"trainer/global_step2:JPxoû¶@°b; loss/FBt3/detection_loss"trainer/global_step2:JP¤ÅÖé@±b; loss/FRt4/detection_loss"trainer/global_step2:JPéÌW-²b( epoch"trainer/global_step2:JPP…A³b< loss/FRt1/assignment_loss"trainer/global_step2:JPÕ›rÿ@´b; loss/FRt1/detection_loss"trainer/global_step2:JP}†Xå@µb; loss/FRt2/detection_loss"trainer/global_step2:JP1ôxˆA¶b< loss/FRt3/assignment_loss"trainer/global_step2:JPUÀ|ƒA·b< loss/FRt2/assignment_loss"trainer/global_step2:JPmðƒ«C¸b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP'ÈC¹b> loss/SRqqt4/assignment_loss"trainer/global_step2:JP¾ÛÙ‚Aºb< loss/FBt1/assignment_loss"trainer/global_step2:JPˆÃÿÞA»b< loss/FBt3/assignment_loss"trainer/global_step2:JP‘Ÿ»Ò@¼b; loss/FRt3/detection_loss"trainer/global_step2:JPðÍÞ@B½b= loss/SRqqt2/detection_loss"trainer/global_step2:JPJÌ‚A¾b< loss/FRt4/assignment_loss"trainer/global_step2:JP âÐB¿b= loss/SRqqt1/detection_loss"trainer/global_step2:JP(ræ¼7Àb2 loss/total_loss"trainer/global_step2:JPIfóþAÁb< loss/FBt4/assignment_loss"trainer/global_step2:JPTÌBÂb= loss/SRqqt4/detection_loss"trainer/global_step2:JPò+`m@Ãb; loss/FBt4/detection_loss"trainer/global_step2:JP™büòCÄb> loss/SRqqt2/assignment_loss"trainer/global_step2:JPz‚ðAÅb< loss/FBt2/assignment_loss"trainer/global_step2:JPƒuÄ{@Æb; loss/FBt1/detection_loss"trainer/global_step2:JP[¾‰0Ç+ _wandbruntime‚106  _runtime‚106›g¹GÈÁ 1loss/SRqqt2/assignment_loss‚5.543277636170387 /loss/FBt2/assignment_loss‚2.695158771106175 /loss/FBt1/detection_loss‚10.499516319483519 1loss/SRqqt1/assignment_loss‚5.427697106411583 0loss/SRqqt3/detection_loss‚6.822710720300674 /loss/FBt2/detection_loss‚12.637816776122365 -loss/FBt3/detection_loss‚2.97085718310515 .loss/FRt4/detection_loss‚2.496735210350204 epoch‚0 /loss/FRt1/assignment_loss‚7.084740602053129 .loss/FRt1/detection_loss‚4.622476863177733 /loss/FRt2/detection_loss‚3.4868310306966306 /loss/FRt3/assignment_loss‚7.413918128380408 trainer/global_step‚49 _step‚3 /loss/FRt2/assignment_loss‚7.360296382904052 1loss/SRqqt3/assignment_loss‚5.391727924346924 1loss/SRqqt4/assignment_loss‚5.349604341718885 /loss/FBt1/assignment_loss‚2.584059491753578 /loss/FBt3/assignment_loss‚2.635281549559699 .loss/FRt3/detection_loss‚4.480692296456068 1loss/SRqqt2/detection_loss‚1.9290713777008932 /loss/FRt4/assignment_loss‚7.213931029842746 0loss/SRqqt1/detection_loss‚4.514964277807035 &loss/total_loss‚125.28943508613878 0loss/FBt4/assignment_loss‚2.6952161697240977 1loss/SRqqt4/detection_loss‚2.6288988159762487 .loss/FBt4/detection_loss‚6.803955080990608 % _timestamp‚1.7885353116555002e+09 _runtime‚ 106.96650648äÃPKÉÅ /loss/FRt3/assignment_loss‚7.413918128380408 trainer/global_step‚49 _step‚3 /loss/FRt2/assignment_loss‚7.360296382904052 1loss/SRqqt3/assignment_loss‚5.391727924346924 1loss/SRqqt4/assignment_loss‚5.349604341718885 /loss/FBt1/assignment_loss‚2.584059491753578 /loss/FBt3/assignment_loss‚2.635281549559699 .loss/FRt3/detection_loss‚4.480692296456068 1loss/SRqqt2/detection_loss‚1.9290713777008932 /loss/FRt4/assignment_loss‚7.213931029842746 0loss/SRqqt1/detection_loss‚4.514964277807035 &loss/total_loss‚125.28943508613878 0loss/FBt4/assignment_loss‚2.6952161697240977 1loss/SRqqt4/detection_loss‚2.6288988159762487 .loss/FBt4/detection_loss‚6.803955080990608 % _timestamp‚1.7885353116555002e+09 _runtime‚ 106.96650648 1loss/SRqqt2/assignment_loss‚5.543277636170387 /loss/FBt2/assignment_loss‚2.695158771106175 /loss/FBt1/detection_loss‚10.499516319483519 1loss/SRqqt1/assignment_loss‚5.427697106411583 0loss/SRqqt3/detection_loss‚6.822710720300674 /loss/FBt2/detection_loss‚12.637816776122365 -loss/FBt3/detection_loss‚2.97085718310515 .loss/FRt4/detection_loss‚2.496735210350204 epoch‚0 /loss/FRt1/assignment_loss‚7.084740602053129 .loss/FRt1/detection_loss‚4.622476863177733 /loss/FRt2/detection_loss‚3.4868310306966306*«ÕV’Ê‚(:‡ œÄëÔæÙÑÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.009' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8018 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+æÈîiË‚(:Þ œÄëÔ¨áûÎ network.sent‚116235 disk.dm-1.out‚ 3.96484375" disk.nvme0c0n1.in‚ 375.23046875 disk.sda3.in‚0$ memory_percent‚95.20399570465088( proc.memory.availableMB‚ 785.77734375 cpu‚25.884550033842732 proc.cpu.threads‚17 disk.nvme0n1.in‚ 375.23046875 disk.sda.out‚ 3.96484375 disk.sda1.out‚0" proc.memory.rssMB‚ 2956.4453125* proc.memory.percent‚18.044710159301758" disk.dm-0.out‚2314.51806640625 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.015625 disk.sda3.out‚ 3.96484375 disk.sda1.in‚0 network.recv‚130720) disk./.usagePercent‚79.04168822591448% disk./.usageGB‚1177.7782745361328 disk.dm-0.in‚ 375.23046875 disk.sda2.out‚0' disk.nvme0c0n1.out‚2314.51806640625% disk.nvme0n1.out‚2314.51806640625 disk.sda.in‚0 þn žÌ‚(:“ «ÄëÔŠ‹•Ì gpu.0.gpu‚21.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.779' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.155799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cy?Í‚(:´ «ÄëÔ—È°Ì disk.nvme0n1.in‚ 375.23046875 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 375.23046875 proc.cpu.threads‚17) disk./.usagePercent‚79.04011326756384 disk.dm-1.in‚0 disk.nvme0c0n1.out‚2651.5" proc.memory.rssMB‚ 2956.4453125 network.recv‚136792 disk.dm-1.out‚4.3125" disk.nvme0c0n1.in‚ 375.23046875 disk.nvme0n1.out‚2651.5 disk.sda3.out‚4.3125 disk.sda.out‚4.3125 disk.sda1.in‚0 disk.dm-0.out‚2651.5% disk./.usageGB‚1177.7548065185547 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚95.20609378814697( proc.memory.availableMB‚ 785.43359375* proc.memory.percent‚18.044710159301758 cpu‚25.838888910343453 network.sent‚166536 disk.sda2.in‚0 disk.drbd1260.out‚0.0156251Ð ¢“΂(:ˆ ºÄëÔø«éË gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚34.0 gpu.0.powerWatts‚173.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“}eÏ‚(:Ú ºÄëÔíÚÂÎ network.recv‚140650 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.01953125" disk.nvme0c0n1.in‚ 375.23046875 disk.nvme0n1.in‚ 375.23046875 disk.dm-0.in‚ 375.23046875' proc.memory.availableMB‚ 785.1328125$ disk./.usageGB‚1177.778408050537 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 4.76171875 disk.dm-1.in‚0" proc.memory.rssMB‚ 2956.4453125* proc.memory.percent‚18.044710159301758$ disk.nvme0n1.out‚3145.8505859375) disk./.usagePercent‚79.04169718617818 disk.sda2.out‚0& disk.nvme0c0n1.out‚3145.8505859375 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚3145.8505859375$ memory_percent‚95.20792961120605 network.sent‚172009 disk.dm-1.out‚ 4.76171875 cpu‚25.79878904090629 proc.cpu.threads‚17 disk.sda3.out‚ 4.76171875“G ¡Ð‚(:… ÉÄëÔ±èÿÊ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.345' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.669 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ŠåËcÑ‚(:Ø ÉÄëÔ¤é¡Ì cpu‚25.765462549525534 network.recv‚145075) disk./.usagePercent‚79.04169488211036 disk.sda1.in‚0 disk.dm-0.in‚ 375.23046875 disk.drbd1260.in‚0& disk.nvme0c0n1.out‚3208.4990234375$ disk.nvme0n1.out‚3208.4990234375 network.sent‚177725 disk.sda.out‚ 5.55859375 disk.dm-1.out‚ 5.55859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 375.23046875$ memory_percent‚95.20721435546875" proc.memory.availableMB‚785.25% disk./.usageGB‚1177.7783737182617! disk.dm-0.out‚3208.4990234375 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 375.23046875# proc.memory.rssMB‚ 2956.44921875* proc.memory.percent‚18.044734001159668 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 5.55859375 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.01953125°£”“Ò‚(:ˆ ØÄëÔמßÊ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.037' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8074 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âD­®dÓ‚(:Ù ØÄëÔú€µÍ proc.cpu.threads‚17 network.sent‚184122 disk.sda2.out‚0' disk.nvme0c0n1.out‚3482.36962890625 disk.sda3.out‚ 5.84765625 disk.sda1.out‚0" disk.dm-0.out‚3482.36962890625 disk.dm-1.in‚0( proc.memory.availableMB‚ 788.46484375 network.recv‚150166 disk.drbd1260.out‚ 0.0234375! disk.nvme0c0n1.in‚ 514.2890625 disk.nvme0n1.in‚ 514.2890625 disk.sda.out‚ 5.84765625 disk.sda1.in‚0$ memory_percent‚95.18759250640869 cpu‚25.73243544379201 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 5.84765625# proc.memory.rssMB‚ 2956.44921875( disk./.usagePercent‚79.0401212037974# disk./.usageGB‚1177.75492477417% disk.nvme0n1.out‚3482.36962890625 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 514.2890625* proc.memory.percent‚18.044734001159668c¬Ô‚(:’ çÄëÔ€ÛëÊ gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.131200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"oÎÎ`Õ‚(:Õ çÄëÔ˜ÇßÍ disk.drbd1260.in‚0' disk.nvme0c0n1.out‚3889.77978515625 disk.sda.in‚0 disk.sda.out‚6.234375 disk.sda1.in‚0# memory_percent‚95.1824426651001 disk.sda3.out‚6.234375 disk.dm-0.in‚ 514.2890625( proc.memory.availableMB‚ 789.30859375) proc.memory.percent‚18.04511547088623 network.sent‚191080 disk.sda2.in‚0 disk.drbd1260.out‚ 0.0234375 disk.nvme0n1.in‚ 514.2890625 disk.sda3.in‚0 disk.sda1.out‚0" disk.dm-0.out‚3889.77978515625# proc.memory.rssMB‚ 2956.51171875) disk./.usagePercent‚79.04011992375973 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 514.2890625% disk.nvme0n1.out‚3889.77978515625 disk.dm-1.in‚0 disk.dm-1.out‚6.234375 cpu‚25.707764236367336 proc.cpu.threads‚17 network.recv‚155740% disk./.usageGB‚1177.7549057006836y“àl0Ö+ _wandbruntime‚201  _runtime‚201ƒ¸¼o×¾ trainer/global_step‚99 % _timestamp‚1.7885354064343932e+09 _runtime‚ 201.746845812 _step‚õL WU4 ' lr-AdamW/pg1‚0.00013801801226886025 ' lr-AdamW/pg2‚0.00013801801226886025E¯ÚÈØ ' lr-AdamW/pg1‚0.00013801801226886025 ' lr-AdamW/pg2‚0.00013801801226886025 trainer/global_step‚99 % _timestamp‚1.7885354064343932e+09 _runtime‚ 201.746845812 _step‚4©òÔT0Ù+ _wandbruntime‚201  _runtime‚201krJë­'Ú§O VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8185747381760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2480025 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2026794 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9043560882688 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 111808325120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1213548 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _runtime‚ 201.752815112 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2480025 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1211 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 112717529600 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2504871 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 112717529600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8185765384704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 117454856704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8034981472136 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 112295369368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8185747381760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9161015739392 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 117690697216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8297555706880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1213550 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 _step‚5 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9043584822784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8297555706880 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚577939 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚99 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2026794 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8298482914304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2026792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7923594070224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8035906478952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 111808325120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8298482914304 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1790883 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 111387401912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2026792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846kÖß±'Û«O KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9043584822784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8297555706880 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚577939 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚99 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2026794 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8298482914304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2026792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7923594070224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8035906478952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 111808325120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8298482914304 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1790883 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 111387401912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2026792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8185747381760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235840512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2480025 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2026794 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9043560882688 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 111808325120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1213548 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _runtime‚ 201.752815112 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2480025 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1211 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 112717529600 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2504871 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 112717529600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8185765384704 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 117454856704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8034981472136 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 112295369368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8185747381760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9161015739392 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 117690697216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8297555706880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1213550 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 _step‚5,«Ø 0Ü+ _wandbruntime‚204  _runtime‚204µª!!·&ݱM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1225613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2529728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9133810527232 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2529728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4551808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8381223473664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8116041346982 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8380296252928 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚583083 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8381223473664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8267366467584 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 118635048448 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8115116339958 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885354089131238e+09 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9252668819968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8267366467584 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1174 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8002628704128 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113839003136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 112504675190 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2046952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚583683 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225615 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9252421635584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9133786587136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2046954 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 118858292736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 _runtime‚ 204.230158819 _step‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2046952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1211 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1809298 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚99 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1808696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8267384470528 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112929785344 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4576682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885354089131238e+09 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9252668819968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8267366467584 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1174 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8002628704128 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113839003136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 112504675190 Æÿ1ã…I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2046952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚583683 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225615 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9252421635584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9133786587136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2046954 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 118858292736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 _runtime‚ 204.230158819 _step‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2046952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1211 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1809298 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚99 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1808696 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8267384470528 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112929785344 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4576682 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2529728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9133810527232 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2529728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4551808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8381223473664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223244288 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8116041346982 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8380296252928 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚583083 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096ƒ¶S¶0ß+ _wandbruntime‚204  _runtime‚204³„1”NàÈ 2loss/SRqqt2/assignment_loss‚4.8696279684702555 /loss/FRt2/detection_loss‚3.3966461857780814 .loss/FBt3/detection_loss‚6.124269056419273 /loss/FBt4/detection_loss‚2.4417697394202493 &loss/total_loss‚112.43622562680365 0loss/FBt2/assignment_loss‚0.6627826454738776 0loss/FBt3/assignment_loss‚0.8440093219280242 epoch‚0 % _timestamp‚1.7885354089255364e+09 /loss/FRt2/assignment_loss‚7.464191883802414 1loss/SRqqt1/assignment_loss‚4.121009579411259 1loss/SRqqt4/assignment_loss‚4.047814141149106 .loss/FRt1/detection_loss‚6.478549052687252 1loss/FBt4/assignment_loss‚0.38381544191687333 .loss/FBt1/detection_loss‚8.700178005977682 /loss/FBt2/detection_loss‚14.221224817136923 trainer/global_step‚99 0loss/SRqqt2/detection_loss‚3.679017382183868 /loss/FRt1/assignment_loss‚7.293506257674274 0loss/SRqqt3/detection_loss‚2.936470836671916 _step‚7 /loss/FRt4/assignment_loss‚7.219080183241102 0loss/FBt1/assignment_loss‚0.5490564386495144 .loss/FRt3/detection_loss‚4.694338306358882 /loss/FRt4/detection_loss‚2.4825349567191943 0loss/SRqqt1/detection_loss‚6.372482090084641 _runtime‚ 204.237483424 /loss/FRt3/assignment_loss‚7.386794362749372 2loss/SRqqt3/assignment_loss‚4.0296648112210365 0loss/SRqqt4/detection_loss‚2.0373921616785775 {RáÌ 0loss/SRqqt2/detection_loss‚3.679017382183868 /loss/FRt1/assignment_loss‚7.293506257674274 0loss/SRqqt3/detection_loss‚2.936470836671916 _step‚7 /loss/FRt4/assignment_loss‚7.219080183241102 0loss/FBt1/assignment_loss‚0.5490564386495144 .loss/FRt3/detection_loss‚4.694338306358882 /loss/FRt4/detection_loss‚2.4825349567191943 0loss/SRqqt1/detection_loss‚6.372482090084641 _runtime‚ 204.237483424 /loss/FRt3/assignment_loss‚7.386794362749372 2loss/SRqqt3/assignment_loss‚4.0296648112210365 0loss/SRqqt4/detection_loss‚2.037392161678577 2loss/SRqqt2/assignment_loss‚4.8696279684702555 /loss/FRt2/detection_loss‚3.3966461857780814 .loss/FBt3/detection_loss‚6.124269056419273 /loss/FBt4/detection_loss‚2.4417697394202493 &loss/total_loss‚112.43622562680365 0loss/FBt2/assignment_loss‚0.6627826454738776 0loss/FBt3/assignment_loss‚0.8440093219280242 epoch‚0 % _timestamp‚1.7885354089255364e+09 /loss/FRt2/assignment_loss‚7.464191883802414 1loss/SRqqt1/assignment_loss‚4.121009579411259 1loss/SRqqt4/assignment_loss‚4.047814141149106 .loss/FRt1/detection_loss‚6.478549052687252 1loss/FBt4/assignment_loss‚0.38381544191687333 .loss/FBt1/detection_loss‚8.700178005977682 /loss/FBt2/detection_loss‚14.221224817136923 trainer/global_step‚99 kÎÔ“â‚(:ˆ öÄëÔè­üÌ gpu.0.gpu‚18.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.009' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8018 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚‹û!^ã‚(:Ó öÄëÔ±æÓÛ$ memory_percent‚95.18415927886963( proc.memory.availableMB‚ 789.02734375 proc.cpu.threads‚17 network.sent‚196582 network.recv‚160045) disk./.usagePercent‚79.04012811600082 disk.sda3.in‚0 disk.sda3.out‚ 6.6328125 disk.sda1.out‚0 disk.dm-1.out‚ 6.6328125 disk.drbd1260.out‚ 0.02734375! disk.nvme0c0n1.in‚ 514.2890625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2956.51171875) proc.memory.percent‚18.04511547088623 cpu‚25.684140322388966% disk.nvme0c0n1.out‚4120.955078125 disk.sda.in‚0 disk.sda.out‚ 6.6328125 disk.dm-0.in‚ 514.2890625$ disk./.usageGB‚1177.755027770996 disk.nvme0n1.in‚ 514.2890625# disk.nvme0n1.out‚4120.955078125 disk.dm-0.out‚4120.955078125 disk.drbd1260.in‚0â©€ä‚(:’ …ÅëÔ÷¯Ë gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:zK]å‚(:Ò …ÅëÔñ“ËÎ network.sent‚245797 network.recv‚165227$ disk./.usageGB‚1177.778476715088 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 7.1328125 disk.sda1.in‚0 disk.sda1.out‚0( disk./.usagePercent‚79.0417017943138 disk.drbd1260.in‚0! disk.nvme0c0n1.in‚ 514.2890625# disk.nvme0n1.out‚4614.845703125 disk.dm-1.in‚0 disk.dm-1.out‚ 7.1328125$ memory_percent‚95.18358707427979# proc.memory.rssMB‚ 2956.51171875 disk.drbd1260.out‚ 0.02734375 disk.sda3.out‚ 7.1328125 disk.sda.in‚0 disk.dm-0.in‚ 514.2890625 disk.dm-0.out‚4614.845703125 proc.cpu.threads‚17% disk.nvme0c0n1.out‚4614.845703125 disk.nvme0n1.in‚ 514.2890625 disk.sda3.in‚0( proc.memory.availableMB‚ 789.12109375) proc.memory.percent‚18.04511547088623 cpu‚25.664684722981704º¡g„Yæ‚(:Î ”ÅëÔ¨€ßË disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 7.68359375( proc.memory.availableMB‚ 788.61328125 proc.cpu.threads‚17$ disk./.usageGB‚1177.778476715088 disk.dm-0.in‚ 621.39453125 disk.dm-1.in‚0 disk.dm-1.out‚ 7.68359375 disk.drbd1260.out‚0.03125" disk.nvme0c0n1.in‚ 621.39453125 disk.sda3.in‚0) proc.memory.percent‚18.04511547088623( disk./.usagePercent‚79.0417017943138 disk.sda1.in‚0 disk.dm-0.out‚ 4615.4140625 disk.drbd1260.in‚0# disk.nvme0c0n1.out‚ 4615.4140625 disk.nvme0n1.in‚ 621.39453125 disk.sda3.out‚ 7.68359375 cpu‚25.646238102851743 network.sent‚252193 disk.sda1.out‚0 disk.sda2.out‚0! disk.nvme0n1.out‚ 4615.4140625# memory_percent‚95.1866865158081# proc.memory.rssMB‚ 2956.51171875 network.recv‚170212í•+žç‚(:“ ”ÅëÔ­ËÊÍ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.939' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.587799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒüdžè‚(:“ £ÅëÔÆ¥ùÊ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@±‘¸Ré‚(:Ç £ÅëÔνåÍ disk.dm-1.out‚ 8.13671875$ memory_percent‚95.18368244171143 network.sent‚259322( disk./.usagePercent‚79.0385518776125% disk./.usageGB‚1177.7315406799316 disk.sda2.in‚0 disk.nvme0n1.out‚ 4917.0625 disk.sda3.in‚0 disk.sda.out‚ 8.13671875 disk.sda1.out‚0 disk.drbd1260.out‚0.03125" disk.nvme0c0n1.in‚ 621.39453125 disk.sda3.out‚ 8.13671875 disk.dm-0.in‚ 621.39453125 disk.dm-1.in‚0( proc.memory.availableMB‚ 789.10546875# proc.memory.rssMB‚ 2956.51171875 cpu‚25.630407179846834 network.recv‚175989 disk.nvme0c0n1.out‚ 4917.0625 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.percent‚18.04511547088623 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 621.39453125 disk.dm-0.out‚ 4917.0625àó¼bê‚(:’ ²ÅëÔõü½Ê gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.177' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.235400000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã’þáeë‚(:Ú ²ÅëÔ‘›ËÌ disk.sda2.out‚0 disk.dm-0.in‚ 621.39453125 disk.dm-1.in‚0 disk.dm-1.out‚ 8.70703125) proc.memory.percent‚18.04516315460205 cpu‚25.615043167249762" disk.nvme0c0n1.in‚ 621.39453125% disk.nvme0n1.out‚5124.83056640625 disk.sda3.in‚0 disk.sda.out‚ 8.70703125% proc.memory.availableMB‚ 789.09375% disk./.usageGB‚1177.7785987854004 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.03125 disk.nvme0n1.in‚ 621.39453125 disk.sda.in‚0$ memory_percent‚95.18375396728516 proc.cpu.threads‚17 network.sent‚264704 network.recv‚179848' disk.nvme0c0n1.out‚5124.83056640625 disk.sda3.out‚ 8.70703125 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚5124.83056640625# proc.memory.rssMB‚ 2956.51953125) disk./.usagePercent‚79.04170998655489[‘Óžì‚(:“ ÁÅëÔÒŽÓÊ gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.205' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.041000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ÀßÓcí‚(:Ø ÁÅëÔ৫Í# proc.memory.rssMB‚ 2956.52734375 cpu‚25.60149727487825$ disk./.usageGB‚1177.755054473877 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 621.39453125' disk.nvme0c0n1.out‚5600.37353515625 disk.sda.in‚0( proc.memory.availableMB‚ 789.08203125 network.sent‚269313 network.recv‚182948 disk.dm-1.in‚0 disk.dm-1.out‚9.203125 disk.drbd1260.out‚ 0.03515625 disk.sda3.in‚0 disk.sda3.out‚9.203125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 621.39453125 disk.sda.out‚9.203125 disk.dm-0.in‚ 621.39453125$ memory_percent‚95.18382549285889) proc.memory.percent‚18.04521083831787 proc.cpu.threads‚17) disk./.usagePercent‚79.04012990805356 disk.sda2.in‚0% disk.nvme0n1.out‚5600.37353515625 disk.sda1.out‚0" disk.dm-0.out‚5600.37353515625èº*žî‚(:“ ÐÅëÔìäüÊ gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚34.0 gpu.0.powerWatts‚164.591' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.918200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Åê~;aï‚(:Ö ÐÅëÔÅîúÌ network.recv‚187499 disk.sda2.in‚0# disk.nvme0n1.out‚6094.513671875 disk.dm-0.in‚ 744.53515625 disk.dm-1.out‚ 9.5078125$ disk./.usageGB‚1177.755054473877" disk.nvme0c0n1.in‚ 744.53515625% disk.nvme0c0n1.out‚6094.513671875 disk.sda3.out‚ 9.5078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚6094.513671875$ memory_percent‚95.17843723297119) disk./.usagePercent‚79.04012990805356 disk.drbd1260.out‚ 0.03515625 disk.nvme0n1.in‚ 744.53515625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 9.5078125( proc.memory.availableMB‚ 789.96484375) proc.memory.percent‚18.04521083831787 network.sent‚275705 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2956.52734375 cpu‚25.588574821341766 proc.cpu.threads‚17åÔ0ð+ _wandbruntime‚301  _runtime‚301»H+ÀÃñ½ & lr-AdamW/pg1‚0.0002077240790713149 & lr-AdamW/pg2‚0.0002077240790713149 trainer/global_step‚149 % _timestamp‚1.7885355061314003e+09 _runtime‚ 301.442580332 _step‚8ä·í¼ÇòÁ trainer/global_step‚149 % _timestamp‚1.7885355061314003e+09 _runtime‚ 301.442580332 _step‚8 & lr-AdamW/pg1‚0.0002077240790713149 & lr-AdamW/pg2‚0.0002077240790713149J_§0ó+ _wandbruntime‚301  _runtime‚301x³!¾'ô¸O D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6759719 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12435524630528 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2682879 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227451904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3724927 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13554846105088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3724927 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 168804954624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12266701672960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 167895750144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚866690 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251392000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12434597423104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 168172624476 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12266719675904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11874490836784 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚866081 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12266719675904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11874473797424 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 167264657020 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13730818956800 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13730567564800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6784567 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12042663461260 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1816798 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚9 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12266701672960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 175948911616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 168804954624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3034794 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3749773|s ‡ õ¼O @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚9 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12266701672960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 175948911616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 168804954624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3034794 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3749773 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6759719 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12435524630528 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2682879 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227451904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3724927 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13554846105088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3724927 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 168804954624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12266701672960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 167895750144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚866690 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251392000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12434597423104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 168172624476 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12266719675904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11874490836784 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚866081 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12266719675904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11874473797424 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 167264657020 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13730818956800 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13730567564800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6784567 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12042663461260 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1816798 'O *0ö+ _wandbruntime‚303  _runtime‚303§¿P_É&÷ÃM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3054954 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 169289897962 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3054954 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 trainer/global_step‚149 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11953508431328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13821965541376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13645071809536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3749758 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚10 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 169017210368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11953491391968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 169926428160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12348338761728 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2700635 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚871772 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6829584 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6804710 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12348320758784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1828865 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚872390 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 168381930298 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13822219017216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12518265189888 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 169926428160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12122798329290 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2701255 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12348320758784 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 169017210368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12121873322266 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3749758 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6829584 _runtime‚ 303.734047052 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 176893731840 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6804710 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3054952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13645095749632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885355084212265e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12517337969152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615^yšoÍ&øÇM MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3054954 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 169289897962 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3054954 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 trainer/global_step‚149 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11953508431328 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13821965541376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13645071809536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3749758 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚10 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 169017210368 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11953491391968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 169926428160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12348338761728 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2700635 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚871772 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6829584 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6804710 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12348320758784 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1828865 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚872390 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 168381930298 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13822219017216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12518265189888 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 169926428160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12122798329290 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2701255 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚12348320758784 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 169017210368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12121873322266 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3749758 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6829584 _runtime‚ 303.734047052 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 176893731840 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6804710 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3054952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13645095749632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885355084212265e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12517337969152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ÷.=0ù+ _wandbruntime‚303  _runtime‚303­*ROúÉ 1loss/SRqqt4/detection_loss‚1.8050231258510738 /loss/FRt2/assignment_loss‚7.445332092709012 0loss/FBt2/assignment_loss‚0.2929706883340892 0loss/SRqqt2/detection_loss‚2.957889864915142 .loss/FRt1/detection_loss‚4.643412888050079 /loss/FRt4/detection_loss‚2.2364768677278226 .loss/FBt2/detection_loss‚7.224677484968434 epoch‚0 % _timestamp‚1.7885355084254668e+09 /loss/FRt1/assignment_loss‚7.239145188104539 2loss/SRqqt1/assignment_loss‚3.3308714539916426 1loss/SRqqt2/assignment_loss‚3.871483891618018 .loss/FBt1/detection_loss‚51.42698857188225 /loss/FBt4/detection_loss‚1.5055319445072524 %loss/total_loss‚150.6267446709994 _step‚11 /loss/FRt3/assignment_loss‚7.330983336766561 1loss/SRqqt4/assignment_loss‚3.340940124279744 /loss/FBt1/assignment_loss‚0.758756955464681 0loss/FBt4/assignment_loss‚0.2583653762567631 .loss/FRt3/detection_loss‚5.517787612477938 0loss/SRqqt1/detection_loss‚3.047629942496618 _runtime‚ 303.736924746 0loss/FRt4/assignment_loss‚7.1892147280953145 0loss/SRqqt3/detection_loss‚4.140533178299665 trainer/global_step‚149 /loss/FRt2/detection_loss‚3.7753724962472917 2loss/SRqqt3/assignment_loss‚3.4537455677986144 0loss/FBt3/assignment_loss‚1.0707490026950837 /loss/FBt3/detection_loss‚16.762862287461758Í`vëSûÍ 1loss/SRqqt4/assignment_loss‚3.340940124279744 /loss/FBt1/assignment_loss‚0.758756955464681 0loss/FBt4/assignment_loss‚0.2583653762567631 .loss/FRt3/detection_loss‚5.517787612477938 0loss/SRqqt1/detection_loss‚3.047629942496618 _runtime‚ 303.736924746 0loss/FRt4/assignment_loss‚7.1892147280953145 0loss/SRqqt3/detection_loss‚4.140533178299665 trainer/global_step‚149 /loss/FRt2/detection_loss‚3.7753724962472917 2loss/SRqqt3/assignment_loss‚3.4537455677986144 0loss/FBt3/assignment_loss‚1.0707490026950837 /loss/FBt3/detection_loss‚16.762862287461758 1loss/SRqqt4/detection_loss‚1.8050231258510738 /loss/FRt2/assignment_loss‚7.445332092709012 0loss/FBt2/assignment_loss‚0.2929706883340892 0loss/SRqqt2/detection_loss‚2.957889864915142 .loss/FRt1/detection_loss‚4.643412888050079 /loss/FRt4/detection_loss‚2.2364768677278226 .loss/FBt2/detection_loss‚7.224677484968434 epoch‚0 % _timestamp‚1.7885355084254668e+09 /loss/FRt1/assignment_loss‚7.239145188104539 2loss/SRqqt1/assignment_loss‚3.3308714539916426 1loss/SRqqt2/assignment_loss‚3.871483891618018 .loss/FBt1/detection_loss‚51.42698857188225 /loss/FBt4/detection_loss‚1.5055319445072524 %loss/total_loss‚150.6267446709994 _step‚11 /loss/FRt3/assignment_loss‚7.330983336766561 Î(`“‘ü‚(:† ßÅëÔËžßÊ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.455' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.691 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼Egj`ý‚(:Õ ßÅëÔá»ÞÖ disk.sda3.in‚0( disk./.usagePercent‚79.0417048664042$ disk./.usageGB‚1177.778522491455" disk.nvme0c0n1.in‚ 744.53515625 disk.nvme0n1.in‚ 744.53515625 disk.sda1.out‚0 disk.dm-0.in‚ 744.53515625 disk.dm-0.out‚6229.333984375 network.recv‚192342 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 10.39453125$ memory_percent‚95.18046379089355' proc.memory.availableMB‚ 789.6328125 proc.memory.rssMB‚ 2956.53125 network.sent‚282147 disk.drbd1260.out‚ 0.0390625% disk.nvme0c0n1.out‚6229.333984375# disk.nvme0n1.out‚6229.333984375 disk.sda1.in‚0 disk.dm-1.in‚0) proc.memory.percent‚18.04523468017578 cpu‚25.57593023292121 disk.sda2.in‚0 disk.sda3.out‚ 10.39453125 disk.sda.in‚0 disk.sda.out‚ 10.39453125 proc.cpu.threads‚17<†õs“þ‚(:ˆ îÅëÔÊ´õÊ gpu.0.gpu‚15.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢¢ì\ÿ‚(:Ï îÅëÔîÚ´Ó network.sent‚333207 disk.drbd1260.in‚0 disk.sda.in‚0 cpu‚25.565451543690553 network.recv‚197478" disk.nvme0c0n1.in‚ 744.53515625% disk.nvme0c0n1.out‚6468.568359375 disk.nvme0n1.in‚ 744.53515625# disk.nvme0n1.out‚6468.568359375 disk.sda1.in‚0 disk.dm-0.out‚6468.568359375 disk.dm-1.in‚0% disk./.usageGB‚1177.7785415649414 disk.drbd1260.out‚ 0.0390625 disk.sda3.in‚0 disk.sda3.out‚10.84375 disk.sda.out‚10.84375 disk.dm-0.in‚ 744.53515625 disk.dm-1.out‚10.84375$ memory_percent‚95.17903327_¾7Ûþ941895) disk./.usagePercent‚79.04170614644188 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 789.8671875 proc.memory.rssMB‚ 2956.53125) proc.memory.percent‚18.04523468017578 proc.cpu.threads‚17íï5Íž€‚(:“ ýÅëÔÙÿ¯Ì gpu.0.gpu‚23.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚35.0 gpu.0.powerWatts‚172.635' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.526999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!(8§`‚(:Õ ýÅëÔ¯š¾Î disk.nvme0n1.in‚ 744.53515625 disk.sda.out‚ 11.3515625 disk.dm-1.in‚0 proc.memory.rssMB‚ 2956.53125$ disk./.usageGB‚1177.778522491455 disk.sda1.in‚0# proc.memory.availableMB‚790.125 cpu‚25.55408351789095 proc.cpu.threads‚17( disk./.usagePercent‚79.0417048664042 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 744.53515625' disk.nvme0c0n1.out‚6675.93896484375% disk.nvme0n1.out‚6675.93896484375 disk.sda3.in‚0 disk.sda3.out‚ 11.3515625 disk.dm-0.in‚ 744.53515625 disk.drbd1260.out‚ 0.04296875 disk.sda.in‚0 disk.sda1.out‚0" disk.dm-0.out‚6675.93896484375 disk.dm-1.out‚ 11.3515625$ memory_percent‚95.17745971679688) proc.memory.percent‚18.04523468017578 network.sent‚336897 network.recv‚201681 disk.sda2.in‚0 disk.drbd1260.in‚0iŽÐ“‚‚(:ˆ ŒÆëÔ¸­ÄÊ gpu.0.gpu‚18.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qÿî0\ƒ‚(:Ñ ŒÆëÔ›æÕÑ disk.dm-0.in‚901.3125 disk.sda2.out‚0 disk.sda3.out‚ 11.9609375 disk.sda.out‚ 11.9609375 cpu‚25.54679645649554 proc.cpu.threads‚17 disk.dm-1.out‚ 11.9609375 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚901.3125' disk.nvme0c0n1.out‚7021.53662109375 disk.nvme0n1.in‚901.3125$ memory_percent‚95.17903327941895# proc.memory.rssMB‚ 2956.53515625 network.recv‚206564) disk./.usagePercent‚79.04012990805356 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.04296875% disk.nvme0n1.out‚7021.53662109375 disk.sda.in‚0' proc.memory.availableMB‚ 789.8671875 network.sent‚345012$ disk./.usageGB‚1177.755054473877 disk.sda1.in‚0" disk.dm-0.out‚7021.53662109375 disk.sda3.in‚0) proc.memory.percent‚18.045258522033697à“„‚(:ˆ ›ÆëÔ¢Í÷Ì gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öç3b…‚(:× ›ÆëÔÿ™ËÏ network.recv‚211534 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.046875% disk.nvme0n1.out‚7338.49853515625 disk.sda3.in‚0 network.sent‚349730 disk.dm-0.in‚ 927.796875" disk.dm-0.out‚7338.49853515625# memory_percent‚95.1775312423706( proc.memory.availableMB‚ 790.11328125$ disk./.usageGB‚1177.766773223877 disk.sda.in‚0 disk.sda.out‚ 12.22265625 disk.dm-1.out‚ 12.22265625 disk.nvme0c0n1.in‚ 927.796875' disk.nvme0c0n1.out‚7338.49853515625 disk.nvme0n1.in‚ 927.796875 disk.sda3.out‚ 12.22265625) disk./.usagePercent‚79.04091636319876 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2956.53515625) proc.memory.percent‚18.04525852203369 cpu‚25.5383024760456 proc.cpu.threads‚17ñv!‚‘†‚(:† ªÆëÔþ•ùÊ gpu.0.gpu‚59.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚34.0 gpu.0.powerWatts‚121.16' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@iåÒZ‡‚(:Ï ªÆëÔµÏÅÏ disk.drbd1260.in‚0 disk.nvme0c0n1.in‚ 927.796875% disk.nvme0n1.out‚7684.09228515625" disk.dm-0.out‚7684.09228515625 disk.sda3.out‚12.84375 network.sent‚355164 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚0.046875 disk.sda3.in‚0$ memory_percent‚95.17908096313477& proc.memory.availableMB‚ 789.859375% disk./.usageGB‚1177.7785415649414' disk.nvme0c0n1.out‚7684.09228515625 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚12.84375# proc.memory.rssMB‚ 2956.53515625) proc.memory.percent‚18.04525852203369 cpu‚25.53015954645879 network.recv‚215517) disk./.usagePercent‚79.04170614644188 disk.nvme0n1.in‚ 927.796875 disk.sda.out‚12.84375 disk.sda1.out‚0 disk.dm-0.in‚ 927.796875 disk.dm-1.in‚0 proc.cpu.threads‚17âÎë„0ˆ+ _wandbruntime‚396  _runtime‚396PÊ hʼn¿ $ _timestamp‚1.788535601659729e+09 _runtime‚ 396.971797899 _step‚12 ' lr-AdamW/pg1‚0.00027743014587376956 ' lr-AdamW/pg2‚0.00027743014587376956 trainer/global_step‚199–?^ÉŠÃ ' lr-AdamW/pg1‚0.00027743014587376956 ' lr-AdamW/pg2‚0.00027743014587376956 trainer/global_step‚199 $ _timestamp‚1.788535601659729e+09 _runtime‚ 396.971797899 _step‚12 MÓ`?0‹+ _wandbruntime‚396  _runtime‚396}ü\Á'Œ»O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2420050 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16571639139328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 % _timestamp‚1.7885356016643596e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 223983175168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1156471 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16048495436752 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15825353524624 trainer/global_step‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16347673967104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 224049879584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9037469 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4969829 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16347655964160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 223141912128 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4994675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16347655964160 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18300645368832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237937664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚13 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 234490101248 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18300383491072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 224892379648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16571639139328 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4042792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223983175168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3576519 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16572566346752 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 234252163584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2420048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚5691I5ÝÅ'¿O bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 234490101248 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18300383491072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 224892379648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16571639139328 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4042792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223983175168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3576519 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚16572566346752 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 234252163584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2420048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1216 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2420050 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16571639139328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 % _timestamp‚1.7885356016643596e+09 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 223983175168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1156471 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16048495436752 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15825353524624 trainer/global_step‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16347673967104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 224049879584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9037469 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4969829 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16347655964160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 223141912128 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4994675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16347655964160 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18300645368832 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237937664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚13 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 ¢x¬û0Ž+ _wandbruntime‚399  _runtime‚399ø•=ÅM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237924352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16654379685376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4062952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9082486 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16128630304574 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9082486 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3594941 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1162826 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18391764729344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18156380972032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚199 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1162201 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2432113 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 225167153070 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3594314 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16429293052928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 225104635392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16655306906112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4994660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261864448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18156357031936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4062952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16129555311598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚14R¾”Ï&ÉM MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4994660 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261864448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18156357031936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4062952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16129555311598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚14 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237924352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16654379685376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4062952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9082486 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16128630304574 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9082486 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3594941 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1162826 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18391764729344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18156380972032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚199 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1162201 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2432113 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 225167153070 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3594314 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16429293052928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 225104635392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16655306906112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508g°½â0‘+ _wandbruntime‚399  _runtime‚399e‚IgN’È _runtime‚ 399.413014232 1loss/SRqqt3/assignment_loss‚3.584486597776413 .loss/FBt2/detection_loss‚8.045675122737885 epoch‚0 _step‚15 0loss/FRt2/assignment_loss‚7.3776459279267685 .loss/FRt3/assignment_loss‚7.41937418282032 /loss/FBt4/assignment_loss‚0.407028488170069 .loss/FRt2/detection_loss‚3.554321842349094 .loss/FRt3/detection_loss‚4.923688905313611 2loss/SRqqt1/assignment_loss‚3.4474706131479014 1loss/SRqqt3/detection_loss‚3.8867678694427013 .loss/FBt1/detection_loss‚9.415862613916397 trainer/global_step‚199 /loss/FRt4/assignment_loss‚7.162777712850859 1loss/SRqqt2/assignment_loss‚3.728027398469018 0loss/SRqqt1/detection_loss‚6.897616337174955 1loss/SRqqt2/detection_loss‚2.0340354364579087 2loss/SRqqt4/assignment_loss‚3.2971455989739833 0loss/FBt1/assignment_loss‚0.8473774459853303 0loss/FBt2/assignment_loss‚0.7075878219545757 .loss/FRt4/detection_loss‚2.196304051374847 % _timestamp‚1.7885356041010208e+09 /loss/FRt1/assignment_loss‚7.123278915882111 1loss/SRqqt4/detection_loss‚1.7010502056815686 .loss/FBt3/detection_loss‚4.928123752276103 0loss/FBt3/assignment_loss‚0.9714863962597318 /loss/FRt1/detection_loss‚3.1271648332476616 /loss/FBt4/detection_loss‚1.0130224868453717 %loss/total_loss‚97.79732055703519R3x—R“Ì 2loss/SRqqt4/assignment_loss‚3.2971455989739833 0loss/FBt1/assignment_loss‚0.8473774459853303 0loss/FBt2/assignment_loss‚0.7075878219545757 .loss/FRt4/detection_loss‚2.196304051374847 % _timestamp‚1.7885356041010208e+09 /loss/FRt1/assignment_loss‚7.123278915882111 1loss/SRqqt4/detection_loss‚1.7010502056815686 .loss/FBt3/detection_loss‚4.928123752276103 0loss/FBt3/assignment_loss‚0.9714863962597318 /loss/FRt1/detection_loss‚3.1271648332476616 /loss/FBt4/detection_loss‚1.0130224868453717 %loss/total_loss‚97.79732055703519 _runtime‚ 399.413014232 1loss/SRqqt3/assignment_loss‚3.584486597776413 .loss/FBt2/detection_loss‚8.045675122737885 epoch‚0 _step‚15 0loss/FRt2/assignment_loss‚7.3776459279267685 .loss/FRt3/assignment_loss‚7.41937418282032 /loss/FBt4/assignment_loss‚0.407028488170069 .loss/FRt2/detection_loss‚3.554321842349094 .loss/FRt3/detection_loss‚4.923688905313611 2loss/SRqqt1/assignment_loss‚3.4474706131479014 1loss/SRqqt3/detection_loss‚3.8867678694427013 .loss/FBt1/detection_loss‚9.415862613916397 trainer/global_step‚199 /loss/FRt4/assignment_loss‚7.162777712850859 1loss/SRqqt2/assignment_loss‚3.728027398469018 0loss/SRqqt1/detection_loss‚6.897616337174955 1loss/SRqqt2/detection_loss‚2.0340354364579087ƒöým“”‚(:ˆ ¹ÆëÔáÛÌ gpu.0.gpu‚15.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.117' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0234 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xcëe•‚(:Ú ¹ÆëÔ¬µ”Ï& proc.memory.availableMB‚ 790.046875 disk.sda1.out‚0" disk.dm-0.out‚7895.45263671875 disk.dm-1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚7895.45263671875 proc.cpu.threads‚17% disk./.usageGB‚1177.7787704467773 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2956.53515625 network.recv‚219680) disk./.usagePercent‚79.04172150689394 disk.sda3.out‚ 13.45703125 disk.dm-0.in‚ 927.796875% disk.nvme0n1.out‚7895.45263671875) proc.memory.percent‚18.04525852203369 cpu‚25.52534747631875 network.sent‚360732 disk.sda.out‚ 13.45703125 disk.dm-1.out‚ 13.45703125 disk.drbd1260.out‚ 0.05078125 disk.nvme0c0n1.in‚ 927.796875 disk.nvme0n1.in‚ 927.796875$ memory_percent‚95.17793655395508°§ìã’–‚(:‡ ÈÆëÔôã³Ë gpu.0.gpu‚4.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍJÐØd—‚(:Ù ÈÆëÔ¸†¤Í disk.sda.in‚0 disk.sda.out‚ 13.87890625 disk.sda1.out‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 927.796875% disk.nvme0n1.out‚8193.14404296875) proc.memory.percent‚18.04525852203369 proc.cpu.threads‚17 network.recv‚225423) disk./.usagePercent‚79.04013221212138" disk.dm-0.out‚8193.14404296875 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.05078125% proc.memory.availableMB‚ 789.65625 disk.sda3.out‚ 13.87890625 disk.sda1.in‚0 disk.dm-0.in‚ 927.796875 disk.dm-1.out‚ 13.87890625 disk.nvme0c0n1.in‚ 927.796875# memory_percent‚95.1803207397461# proc.memory.rssMB‚ 2956.53515625 cpu‚25.520476411860546 network.sent‚410633% disk./.usageGB‚1177.7550888061523 disk.sda3.in‚0 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚8193.14404296875­ M®’˜‚(:‡ ×ÆëÔ°ó»Ê gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚34.0 gpu.0.powerWatts‚164.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.937 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹æò7`™‚(:Õ ×ÆëÔ€…¦Í( proc.memory.availableMB‚ 789.88671875 proc.cpu.threads‚17 network.recv‚230373 disk.sda1.out‚0% disk.nvme0n1.out‚8510.42236328125 disk.sda3.in‚0 disk.sda3.out‚14.15625# proc.memory.rssMB‚ 2956.53515625 network.sent‚417196) disk./.usagePercent‚79.04091866726655 disk.sda2.in‚0 disk.nvme0n1.in‚ 1071.203125 disk.sda.out‚14.15625) proc.memory.percent‚18.04525852203369% disk./.usageGB‚1177.7668075561523" disk.dm-0.out‚8510.42236328125 disk.dm-1.in‚0 disk.dm-1.out‚14.15625! disk.nvme0c0n1.in‚ 1071.203125' disk.nvme0c0n1.out‚8510.42236328125 disk.sda.in‚0 cpu‚25.514150108174743 disk.sda1.in‚0 disk.dm-0.in‚ 1071.203125 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0546875# memory_percent‚95.1789140701294ÂÇ“š‚(:ˆ æÆëÔ¶„ÙË gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.998' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“x†®^›‚(:Ó æÆëÔç¼ÕÌ disk.sda3.out‚14.53125 disk.sda2.in‚0 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚8831.99267578125% disk.nvme0n1.out‚8831.99267578125 network.sent‚424520% disk./.usageGB‚1177.7551078796387 disk.sda3.in‚0 disk.dm-0.in‚ 1071.203125 disk.sda2.out‚0 disk.nvme0n1.in‚ 1071.203125# memory_percent‚95.1789379119873 proc.cpu.threads‚17 network.recv‚234320) disk./.usagePercent‚79.04013349215904 disk.sda.out‚14.53125 disk.sda1.out‚0" disk.dm-0.out‚8831.99267578125 disk.dm-1.out‚14.53125 disk.drbd1260.out‚ 0.0546875' proc.memory.availableMB‚ 789.8828125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 1071.203125# proc.memory.rssMB‚ 2956.53515625) proc.memory.percent‚18.04525852203369 cpu‚25.50977781745961“ÊÈ7‘œ‚(:† õÆëÔ¼ƒŒË gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7826 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÒ‡\g‚(:Ü õÆëÔÚå³Í proc.cpu.threads‚17 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚9063.24658203125 disk.nvme0n1.in‚ 1071.203125 disk.sda3.in‚0 disk.sda.out‚ 14.91015625 disk.sda1.out‚0 disk.dm-0.in‚ 1071.203125 disk.dm-1.in‚0 network.recv‚238460 disk.sda2.out‚0 disk.dm-1.out‚ 14.91015625( proc.memory.percent‚18.0452823638916 cpu‚25.504059044803068 network.sent‚428102' disk./.usagePercent‚79.041703330359% disk./.usageGB‚1177.7784996032715 disk.sda2.in‚0 disk.drbd1260.out‚ 0.05859375 disk.sda3.out‚ 14.91015625( proc.memory.availableMB‚ 789.86328125! disk.nvme0c0n1.in‚ 1071.203125% disk.nvme0n1.out‚9063.24658203125 disk.sda.in‚0 disk.sda1.in‚0" disk.dm-0.out‚9063.24658203125$ memory_percent‚95.17905712127686" proc.memory.rssMB‚ 2956.5390625Eʨžž‚(:“ „ÇëÔÖýßÊ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚36.0 gpu.0.powerWatts‚178.056' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.611200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑèÅgŸ‚(:Ü „ÇëÔ¿àŒÍ% disk.nvme0n1.out‚9374.63720703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚95.18067836761475( proc.memory.availableMB‚ 789.59765625 disk.sda3.in‚0 disk.sda3.out‚ 15.3203125 disk.sda.in‚0 disk.sda.out‚ 15.3203125" disk.dm-0.out‚9374.63720703125 disk.dm-1.out‚ 15.3203125 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 1071.203125 disk.nvme0n1.in‚ 1071.203125 disk.sda2.in‚0 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚9374.63720703125# proc.memory.rssMB‚ 2956.54296875) proc.memory.percent‚18.04530620574951 cpu‚25.49782328311284 network.sent‚434319 network.recv‚243111 disk.dm-0.in‚ 1071.203125 disk.drbd1260.out‚ 0.05859375 proc.cpu.threads‚17) disk./.usagePercent‚79.04170947453983% disk./.usageGB‚1177.7785911560059¸Q^+0 + _wandbruntime‚486  _runtime‚4866œœ+Æ¡À _runtime‚ 486.963593642 _step‚16 ' lr-AdamW/pg1‚0.00034713621267622424 ' lr-AdamW/pg2‚0.00034713621267622424 trainer/global_step‚249 % _timestamp‚1.7885356916517384e+09ª1梀 trainer/global_step‚249 % _timestamp‚1.7885356916517384e+09 _runtime‚ 486.963593642 _step‚16 ' lr-AdamW/pg1‚0.00034713621267622424 ' lr-AdamW/pg2‚0.00034713621267622424LÕ0£+ _wandbruntime‚486  _runtime‚486CPh}X ¤¿O =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6214731 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3023298 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5050794 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22870478477312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20428610255360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20055252419060 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11290371 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20056177425876 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 280070600192 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 279019167236 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5050792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 280979804672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1447711 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22870224988160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22577440489984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19776233251824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20708680855552 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6214731 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4470364 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚ 486.964574131 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 280979804672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20709608062976 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 280070600192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 292808438272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 _step‚17 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 279927134692 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20708680855552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3023300 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20428628258304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20428628258304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3023298 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5050794 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22870478477312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20428610255360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20055252419060 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11290371 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20056177425876 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 280070600192 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 279019167236 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5050792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 280979804672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1447711 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22870224988160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22577440489984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19776233251824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20708680855552 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6214731 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4470364 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚ 486.964574131 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚249 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 280979804672 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20709608062976 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 280070600192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 292808438272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 _step‚17 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 279927134692 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20708680855552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3023300 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20510247344128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 282101278208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5070954 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 % _timestamp‚1.7885356926446338e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4488321 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11335388 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 282101278208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 _runtime‚ 487.956510481 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5070952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 281192060416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20136312293906 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20792348622336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20510247344128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1453586 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1452958 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 _step‚18 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22961885670912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚249 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20791421401600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20510229341184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4488951 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20135387286882 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11310514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 294000426496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19855250846368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6264434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 281192060416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11335388 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 294219476480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11310514 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0/áÈJÓ&¨ÍM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11310514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 294000426496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19855250846368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6264434 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 281192060416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11335388 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 294219476480 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11310514 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚20510247344128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 282101278208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5070954 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 % _timestamp‚1.7885356926446338e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4488321 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11335388 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 282101278208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 _runtime‚ 487.956510481 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5070952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 281192060416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20136312293906 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20792348622336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20510247344128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1453586 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1452958 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 _step‚18 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22961885670912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚249 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20791421401600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20510229341184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4488951 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20135387286882 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¹½è0©+ _wandbruntime‚487  _runtime‚487ÏÇÍ—ªÄ /loss/FRt2/assignment_loss‚7.353130435943603 .loss/FRt3/assignment_loss‚7.40300599266501 .loss/FRt1/detection_loss‚2.646560165617201 /loss/FBt4/detection_loss‚1.5978743162517655 2loss/SRqqt4/assignment_loss‚3.2171307121004378 .loss/FBt1/assignment_loss‚1.02909403479498 0loss/SRqqt3/detection_loss‚3.502850984418115 1loss/SRqqt4/detection_loss‚3.9542205499795577 .loss/FBt3/detection_loss‚9.833281768056063 $loss/total_loss‚97.0016829888225 -loss/FRt2/detection_loss‚3.82436977699399 0loss/FBt4/assignment_loss‚0.4870912748613461 /loss/FRt4/detection_loss‚2.0856740974091195 -loss/FBt1/detection_loss‚+úÿâ³10.9662882934014 1loss/SRqqt2/detection_loss‚1.7374463180787738 % _timestamp‚1.7885356926463048e+09 /loss/FRt1/assignment_loss‚7.084692451688978 /loss/FRt4/assignment_loss‚7.279567318993646 0loss/SRqqt1/detection_loss‚5.143748931586742 .loss/FBt2/detection_loss‚3.963666177501804 epoch‚0 trainer/global_step‚249 _step‚19 2loss/SRqqt2/assignment_loss‚3.6208941811009456 1loss/FBt3/assignment_loss‚0.33056787358453643 .loss/FRt3/detection_loss‚2.960425739779192 _runtime‚ 487.957189632 1loss/SRqqt1/assignment_loss‚3.369521370955876 1loss/SRqqt3/assignment_loss‚3.186864353889643 1loss/FBt2/assignment_loss‚0.42371586916977466ŽÉ“N«È .loss/FRt3/detection_loss‚2.960425739779192 _runtime‚ 487.957189632 1loss/SRqqt1/assignment_loss‚3.369521370955876 1loss/SRqqt3/assignment_loss‚3.186864353889643 1loss/FBt2/assignment_loss‚0.42371586916977466 /loss/FRt2/assignment_loss‚7.353130435943603 .loss/FRt3/assignment_loss‚7.40300599266501 .loss/FRt1/detection_loss‚2.646560165617201 /loss/FBt4/detection_loss‚1.5978743162517655 2loss/SRqqt4/assignment_loss‚3.2171307121004378 .loss/FBt1/assignment_loss‚1.02909403479498 0loss/SRqqt3/detection_loss‚3.502850984418115 1loss/SRqqt4/detection_loss‚3.9542205499795577 .loss/FBt3/detection_loss‚9.833281768056063 $loss/total_loss‚97.0016829888225 -loss/FRt2/detection_loss‚3.82436977699399 0loss/FBt4/assignment_loss‚0.4870912748613461 /loss/FRt4/detection_loss‚2.0856740974091195 -loss/FBt1/detection_loss‚10.9662882934014 1loss/SRqqt2/detection_loss‚1.7374463180787738 % _timestamp‚1.7885356926463048e+09 /loss/FRt1/assignment_loss‚7.084692451688978 /loss/FRt4/assignment_loss‚7.279567318993646 0loss/SRqqt1/detection_loss‚5.143748931586742 .loss/FBt2/detection_loss‚3.963666177501804 epoch‚0 trainer/global_step‚249 _step‚19 2loss/SRqqt2/assignment_loss‚3.6208941811009456 1loss/FBt3/assignment_loss‚0.33056787358453643:¤Í“ž¬‚(:“ “ÇëÔ ý³Ë gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.227' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.045399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Ç°Ï`­‚(:Õ “ÇëÔŠÇ÷Ì disk.sda1.in‚0 disk.dm-0.in‚ 1250.52734375) proc.memory.percent‚18.04530620574951) disk./.usagePercent‚78.99659864293888 disk.drbd1260.out‚0.0625 disk.sda3.in‚0! disk.dm-0.out‚9633.3193359375 disk.dm-1.in‚0 disk.dm-1.out‚16.09375' proc.memory.availableMB‚ 789.5078125# proc.memory.rssMB‚ 2956.54296875 network.recv‚248531$ disk.nvme0n1.out‚9633.3193359375 disk.sda.in‚0 disk.sda.out‚16.09375 disk.sda1.out‚0 cpu‚25.486399581704326% disk./.usageGB‚1177.1064071655273 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda3.out‚16.09375$ memory_percent‚95.18122673034668 proc.cpu.threads‚17 network.sent‚441025 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1250.52734375& disk.nvme0c0n1.out‚9633.3193359375! disk.nvme0n1.in‚ 1250.52734375~ yÚ“®‚(:ˆ ¢ÇëÔ½ø¹Ð gpu.0.gpu‚21.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"\Ãxc¯‚(:Ø ¢ÇëÔñÄ× disk.sda.in‚0( disk./.usagePercent‚78.9950249646259% disk./.usageGB‚1177.0829582214355 disk.sda1.in‚0 disk.dm-0.in‚ 1250.52734375! disk.dm-0.out‚9950.2685546875 disk.drbd1260.out‚0.0625& disk.nvme0c0n1.out‚9950.2685546875! disk.nvme0n1.in‚ 1250.52734375 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 1250.52734375$ memory_percent‚95.17824649810791( proc.memory.availableMB‚ 789.99609375 proc.cpu.threads‚17$ disk.nvme0n1.out‚9950.2685546875 disk.sda3.in‚0 disk.sda3.out‚ 16.796875 disk.sda.out‚ 16.796875# proc.memory.rssMB‚ 2956.54296875) proc.memory.percent‚18.04530620574951 cpu‚25.474157115747175 network.sent‚491957 network.recv‚253338 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 16.796875 disk.sda2.in‚0 disk.sda2.out‚0ìHüN‘°‚(:† ±ÇëÔšó‡Î gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.28' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°bóÁd±‚(:Ù ±ÇëÔ†éáÑ) disk./.usagePercent‚78.99502368458823$ disk.nvme0n1.out‚10317.822265625$ memory_percent‚95.17829418182373) proc.memory.percent‚18.04530620574951 cpu‚25.46355065517896 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚17.28125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.06640625( proc.memory.availableMB‚ 789.98828125# proc.memory.rssMB‚ 2956.54296875 network.sent‚495605% disk./.usageGB‚1177.0829391479492 disk.sda3.in‚0! disk.dm-0.out‚10317.822265625 disk.dm-1.out‚17.28125 disk.sda2.in‚0 disk.drbd1260.in‚0& disk.nvme0c0n1.out‚10317.822265625 network.recv‚257391 disk.sda3.out‚17.28125 disk.sda1.in‚0 disk.dm-0.in‚ 1250.52734375# disk.nvme0c0n1.in‚ 1250.52734375! disk.nvme0n1.in‚ 1250.52734375³õò70²+ _wandbruntime‚537  _runtime‚537ý8a&ij¾ & lr-AdamW/pg1‚0.0004168422794786789 & lr-AdamW/pg2‚0.0004168422794786789 trainer/global_step‚299 % _timestamp‚1.7885357424007523e+09 _runtime‚ 537.711705044 _step‚20¥k È´Â _runtime‚ 537.711705044 _step‚20 & lr-AdamW/pg1‚0.0004168422794786789 & lr-AdamW/pg2‚0.0004168422794786789 trainer/global_step‚299 % _timestamp‚1.7885357424007523e+09ÉÃìò0µ+ _wandbruntime‚537  _runtime‚537 ™’ÐÅ'¶¿O SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3626548 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7459633 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27088725712384 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24062934408184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27088701772288 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23727112979024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 351355807744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27440311069184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 337067229696 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 351585356800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24846649779200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1739560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24509564546560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6058792 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27440057580032 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7459633 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24062009401368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24509564546560 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 336158025216 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5366110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 % _timestamp‚1.7885357424019349e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13518425 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24846649779200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24509582549504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 337067229696 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6058794 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3626550 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6058794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24845722571776 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24509582549504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 334896422344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚ 537.713166378 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23727130018384 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24845722571776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1738924 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 trainer/global_step‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7484479 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13543273 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 _step‚21 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 336158025216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 337067229696 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 351585356800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24846649779200 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1739560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24509564546560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6058792 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27440057580032 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7459633 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚24062009401368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24509564546560 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 336158025216 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5366110 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 % _timestamp‚1.7885357424019349e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13518425 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24846649779200 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24509582549504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 337067229696 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6058794 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3626550 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6058794 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24845722571776 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24509582549504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 334896422344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚ 537.713166378 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23727130018384 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24845722571776 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1738924 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 trainer/global_step‚299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7484479 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13543273 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 _step‚21 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 336158025216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6078952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 338188703232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24142144269190 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13588290 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7484464 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 337279485440 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5383376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1745395 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7484464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6078952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27178951416832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24929390338560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1744763 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7509336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24929390338560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24591201635328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24591183632384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27531702511616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requeÊY˜Ósted_bytes.large_pool.freed‚23806130573568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 337279485440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚299 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23806147612928 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 336013695622 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24591201635328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚24591183632384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 _runtime‚ 538.617089283 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7509336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27531453230080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 338188703232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6078954 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 338188703232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6078954 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6078952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 338188703232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24142144269190 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13588290 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7484464 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 337279485440 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5383376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1745395 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7484464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6078952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27178951416832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24929390338560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1744763 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7509336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24929390338560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚24591201635328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚24591183632384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27531702511616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23806130573568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 337279485440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚299 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23806147612928 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 336013695622 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751#"p0»+ _wandbruntime‚538  _runtime‚538,ï…P¼Ê /loss/FRt3/assignment_loss‚7.352812223656233 0loss/FBt2/assignment_loss‚0.2964359688757859 /loss/FRt4/detection_loss‚1.7860973754897713 0loss/SRqqt1/detection_loss‚5.403556107120081 1loss/SRqqt2/detection_loss‚1.5458010692647346 /loss/FBt2/detection_loss‚1.7778739200338074 0loss/FBt4/assignment_loss‚0.5804795504233841 /loss/FRt3/detection_loss‚3.3403825468795243 1loss/SRqqt3/detection_loss‚2.2166063328040764 epoch‚0 _runtime‚ 538.618411585 _step‚23 /loss/FRt4/assignment_loss‚7.294306588172913 1loss/SRqqt1/assignment_loss‚3.808352459560741 2loss/SRqqt3/assignment_loss‚3.0184264332056046 /loss/FRt2/assignment_loss‚7.428722610473633 2loss/SRqqt2/assignment_loss‚3.5915665975431117 /loss/FBt1/assignment_loss‚0.682820000933134 .loss/FRt2/detection_loss‚2.943298884034157 .loss/FBt1/detection_loss‚7.040708061307669 /loss/FBt4/detection_loss‚2.3210836956216325 trainer/global_step‚299 0loss/FBt3/assignment_loss‚0.6276895288716663 .loss/FRt1/detection_loss‚3.197752135140555 .loss/FBt3/detection_loss‚8.393000304698944 $loss/total_loss‚86.9384812886421 /loss/FRt1/assignment_loss‚7.185551881790161 2loss/SRqqt4/assignment_loss‚3.3534902301099567 1loss/SRqqt4/detection_loss‚1.7516667826308145 % _timestamp‚1.7885357433073864e+09×èT½Î 2loss/SRqqt3/assignment_loss‚3.0184264332056046 /loss/FRt2/assignment_loss‚7.428722610473633 2loss/SRqqt2/assignment_loss‚3.5915665975431117 /loss/FBt1/assignment_loss‚0.682820000933134 .loss/FRt2/detection_loss‚2.943298884034157 .loss/FBt1/detection_loss‚7.040708061307669 /loss/FBt4/detection_loss‚2.3210836956216325 trainer/global_step‚299 0loss/FBt3/assignment_loss‚0.6276895288716663 .loss/FRt1/detection_loss‚3.197752135140555 .loss/FBt3/detection_loss‚8.393000304698944 $loss/total_loss‚86.9384812886421 /loss/FRt1/assignment_loss‚7.185551881790161 2loss/SRqqt4/assignment_loss‚3.3534902301099567 1loss/SRqqt4/detection_loss‚1.7516667826308145 % _timestamp‚1.7885357433073864e+09 /loss/FRt3/assignment_loss‚7.352812223656233 0loss/FBt2/assignment_loss‚0.2964359688757859 /loss/FRt4/detection_loss‚1.7860973754897713 0loss/SRqqt1/detection_loss‚5.403556107120081 1loss/SRqqt2/detection_loss‚1.5458010692647346 /loss/FBt2/detection_loss‚1.7778739200338074 0loss/FBt4/assignment_loss‚0.5804795504233841 /loss/FRt3/detection_loss‚3.3403825468795243 1loss/SRqqt3/detection_loss‚2.2166063328040764 epoch‚0 _runtime‚ 538.618411585 _step‚23 /loss/FRt4/assignment_loss‚7.294306588172913 1loss/SRqqt1/assignment_loss‚3.808352459560741‡lqÛ“¾‚(:ˆ ÀÇëÔµª§Ë gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õRòj¿‚(:ß ÀÇëÔÑ•–Î disk.dm-1.in‚0 disk.dm-1.out‚ 17.84765625 disk.sda2.in‚0$ disk.nvme0n1.out‚10541.048828125 network.recv‚262191# disk.nvme0c0n1.in‚ 1250.52734375& disk.nvme0c0n1.out‚10541.048828125 disk.sda.out‚ 17.84765625 disk.sda1.in‚0) proc.memory.percent‚18.04530620574951 cpu‚25.453759046705 proc.cpu.threads‚17 disk.drbd1260.in‚0) disk./.usagePercent‚78.99503187682933% disk./.usageGB‚1177.0830612182617! disk.dm-0.out‚10518.798828125! disk.nvme0n1.in‚ 1250.52734375 disk.sda3.in‚0 disk.sda3.out‚ 17.84765625 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚95.18170356750488' proc.memory.availableMB‚ 789.4296875# proc.memory.rssMB‚ 2956.54296875 disk.sda2.out‚0 disk.drbd1260.out‚ 0.06640625 network.sent‚501871 disk.dm-0.in‚ 1250.52734375R “Î’À‚(:‡ ÏÇëÔ¹·Ë gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚37.0 gpu.0.powerWatts‚185.535' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.107 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™kÁ‚(:à ÏÇëÔ¾öË disk.sda3.out‚ 18.22265625 disk.sda.in‚0 disk.dm-0.in‚ 1424.265625& proc.memory.availableMB‚ 789.671875 disk.sda2.out‚0& disk.nvme0n1.out‚10870.09033203125 disk.sda1.out‚0$ memory_percent‚95.18022537231445 cpu‚25.444333403929924 network.recv‚268403% disk./.usageGB‚1177.1064758300781 disk.drbd1260.out‚ 0.0703125 disk.sda3.in‚0# proc.memory.rssMB‚ 2956.54296875) proc.memory.percent‚18.04530620574951 proc.cpu.threads‚17 network.sent‚552093 disk.drbd1260.in‚0! disk.nvme0c0n1.in‚ 1424.265625 disk.sda.out‚ 18.22265625 disk.sda1.in‚0# disk.dm-0.out‚10870.09033203125 disk.dm-1.in‚0 disk.dm-1.out‚ 18.22265625) disk./.usagePercent‚78.99660325107449 disk.sda2.in‚0( disk.nvme0c0n1.out‚10870.09033203125 disk.nvme0n1.in‚ 1424.265625± c’‚(:‡ ÞÇëÔ—ê»Ë gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7608 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l^~]aÂ(:Ö ÞÇëÔäèÌ disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.0703125( disk.nvme0c0n1.out‚10870.35205078125 disk.sda.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 789.68359375) disk./.usagePercent‚78.99660453111217% disk./.usageGB‚1177.1064949035645! disk.nvme0c0n1.in‚ 1424.265625& disk.nvme0n1.out‚10870.35205078125 disk.sda3.in‚0 disk.sda.out‚18.8125 disk.sda1.out‚0# disk.dm-0.out‚10870.35205078125 disk.nvme0n1.in‚ 1424.265625 disk.dm-0.in‚ 1424.265625 disk.dm-1.out‚18.8125$ memory_percent‚95.18015384674072# proc.memory.rssMB‚ 2956.54296875) proc.memory.percent‚18.04530620574951 network.sent‚557906 disk.drbd1260.in‚0 disk.sda3.out‚18.8125 disk.dm-1.in‚0 cpu‚25.443249661813283 proc.cpu.threads‚17 network.recv‚273000X™žÄ‚(:“ íÇëÔÑˤÊ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.651' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.130200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šûâÞiÅ‚(:Þ íÇëÔ­Ó³Í( disk.nvme0c0n1.out‚11159.15966796875 disk.nvme0n1.in‚ 1424.265625& disk.nvme0n1.out‚11159.15966796875 disk.sda3.out‚ 19.23046875 disk.sda.in‚0$ proc.memory.availableMB‚789.4375 network.sent‚565129% disk./.usageGB‚1176.9763145446777 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 1424.265625# proc.memory.rssMB‚ 2956.54296875 cpu‚25.441715419802765( disk./.usagePercent‚78.9878680179987 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.07421875! disk.nvme0c0n1.in‚ 1424.265625 disk.sda.out‚ 19.23046875 disk.sda1.in‚0# disk.dm-0.out‚11159.15966796875$ memory_percent‚95.18165588378906 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 19.23046875) proc.memory.percent‚18.04530620574951 proc.cpu.threads‚17 network.recv‚276849 disk.sda2.out‚0%¨ÒnÆ‚(:ã üÇëÔëûÑÌ disk.sda2.out‚0$ memory_percent‚95.18015384674072( proc.memory.availableMB‚ 789.68359375 network.sent‚569483 network.recv‚281472 disk.sda3.in‚0 disk.dm-1.out‚ 19.79296875 disk.sda2.in‚0 cpu‚25.433487751943964 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.07421875 disk.sda3.out‚ 19.79296875 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1424.265625# disk.dm-0.out‚11550.54248046875 disk.dm-1.in‚0) proc.memory.percent‚18.04530620574951) disk./.usagePercent‚78.98778891167059% disk./.usageGB‚1176.9751358032227! disk.nvme0c0n1.in‚ 1424.265625( disk.nvme0c0n1.out‚11550.54248046875& disk.nvme0n1.out‚11550.54248046875 disk.sda.out‚ 19.79296875 disk.sda1.out‚0# proc.memory.rssMB‚ 2956.54296875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 1424.265625jZš’Ç‚(:‡ üÇëÔûëÔÍ gpu.0.gpu‚57.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 É0È+ _wandbruntime‚607  _runtime‚607¦ÈßÄɾ _step‚24 & lr-AdamW/pg1‚0.0004865483462811336 & lr-AdamW/pg2‚0.0004865483462811336 trainer/global_step‚349 % _timestamp‚1.7885358121674485e+09 _runtime‚ 607.478966472â üÈÊ & lr-AdamW/pg1‚0.0004865483462811336 & lr-AdamW/pg2‚0.0004865483462811336 trainer/global_step‚349 % _timestamp‚1.7885358121674485e+09 _runtime‚ 607.478966472 _step‚24¤Ô…¹0Ë+ _wandbruntime‚607  _runtime‚607_¤oâ#Ì¿O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6259371 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 392245450240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2029573 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15771327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28590536840704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4229798 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2030180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6259980 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27678009745584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7066794 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28590536840704 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 393154654720 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28982764288000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 392245450240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7066794 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7066792 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28982764288000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 410037463040 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 391681644908 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 410262817792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28590518837760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31599986994688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 _runtime‚ 607.480544526 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32010273752576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569³4yÉ'ÍÃO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6259371 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 392245450240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2029573 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15771327 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28590536840704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4229798 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2030180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6259980 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27678009745584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7066794 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28590536840704 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 393154654720 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28982764288000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 392245450240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7066794 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7066792 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28982764288000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 410037463040 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 391681644908 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 410262817792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28590518837760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31599986994688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 _runtime‚ 607.480544526 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32010273752576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144KI}0Î+ _wandbruntime‚608  _runtime‚608Æ;¬³Ï&ÏÉM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15841192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2035485 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 _runtime‚ 608.412030102 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚29065504834048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6277348 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28149826258522 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392798918394 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7086954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15816318 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8754238 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8754238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7086952 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29066432054784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28672155926528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672137923584 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 % _timestamp‚1.7885358131002903e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15841192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29066432054784 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32101410941952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28672137923584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8729366 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 394276128256 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 411427778560 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 411198242816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6277967 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6277348 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28149826258522 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392798918394 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7086954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15816318 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8754238 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8754238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7086952 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚29066432054784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28672155926528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672137923584 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 % _timestamp‚1.7885358131002903e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15841192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29066432054784 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32101410941952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28672137923584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8729366 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 394276128256 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 411427778560 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 411198242816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6277967 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15841192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751Ž·Xd0Ñ+ _wandbruntime‚608  _runtime‚608—@€NÒÈ _step‚27 1loss/SRqqt1/assignment_loss‚3.438825424979715 1loss/FBt2/assignment_loss‚0.36477284595405635 %loss/total_loss‚90.26469976601011 trainer/global_step‚349 _runtime‚ 608.413814195 /loss/FRt2/detection_loss‚2.9250694740386236 /loss/FRt4/detection_loss‚2.0559050204465166 0loss/SRqqt1/detection_loss‚4.919792855487151 .loss/FBt2/detection_loss‚4.549841285603387 /loss/FRt1/assignment_loss‚7.285501658916473 epoch‚0 1loss/SRqqt3/assignment_loss‚3.239106746820303 2loss/SRqqt4/assignment_loss‚3.1616391266093533 % _timestamp‚1.7885358131030056e+09 0loss/FBt1/assignment_loss‚0.7656664279944039 /loss/FBt3/assignment_loss‚0.952221468091011 .loss/FBt1/detection_loss‚6.036770093720406 .loss/FRt1/detection_loss‚4.649368761728208 0loss/SRqqt2/detection_loss‚3.289461207058695 1loss/SRqqt4/detection_loss‚0.7473108300884419 -loss/FBt3/detection_loss‚9.21702481508255 /loss/FBt4/detection_loss‚2.0315386553178336 0loss/FRt2/assignment_loss‚7.2460453396751765 /loss/FRt3/assignment_loss‚7.407858079793502 /loss/FRt4/assignment_loss‚7.085585504770279 1loss/SRqqt2/assignment_loss‚3.351493994394938 0loss/FBt4/assignment_loss‚0.6534748242481838 .loss/FRt3/detection_loss‚2.643701995513877 1loss/SRqqt3/detection_loss‚2.2467233296770317p% .Ó$–$NÌ .loss/FBt1/detection_loss‚6.036770093720406 .loss/FRt1/detection_loss‚4.649368761728208 0loss/SRqqt2/detection_loss‚3.289461207058695 1loss/SRqqt4/detection_loss‚0.7473108300884419 -loss/FBt3/detection_loss‚9.21702481508255 /loss/FBt4/detection_loss‚2.0315386553178336 0loss/FRt2/assignment_loss‚7.2460453396751765 /loss/FRt3/assignment_loss‚7.407858079793502 /loss/FRt4/assignment_loss‚7.085585504770279 1loss/SRqqt2/assignment_loss‚3.351493994394938 0loss/FBt4/assignment_loss‚0.6534748242481838 .loss/FRt3/detection_loss‚2.643701995513877 1loss/SRqqt3/detection_loss‚2.2467233296770317 _step‚27 1loss/SRqqt1/assignment_loss‚3.438825424979715 1loss/FBt2/assignment_loss‚0.36477284595405635 %loss/total_loss‚90.26469976601011 trainer/global_step‚349 _runtime‚ 608.413814195 /loss/FRt2/detection_loss‚2.9250694740386236 /loss/FRt4/detection_loss‚2.0559050204465166 0loss/SRqqt1/detection_loss‚4.919792855487151 .loss/FBt2/detection_loss‚4.549841285603387 /loss/FRt1/assignment_loss‚7.285501658916473 epoch‚0 1loss/SRqqt3/assignment_loss‚3.239106746820303 2loss/SRqqt4/assignment_loss‚3.1616391266093533 % _timestamp‚1.7885358131030056e+09 0loss/FBt1/assignment_loss‚0.7656664279944039 /loss/FBt3/assignment_loss‚0.9522214680910112:[™“Ô‚(:ˆ ‹ÈëÔƘ¿Ì gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.218' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ª|lÕ‚(:á ‹ÈëÔòéÏ disk.sda2.in‚0 disk.drbd1260.out‚0.078125# disk.nvme0c0n1.in‚ 1575.63671875! disk.nvme0n1.in‚ 1575.63671875 disk.sda3.out‚ 20.58984375" memory_percent‚95.182204246521( proc.memory.availableMB‚ 789.34765625 disk.dm-0.in‚ 1575.63671875 disk.dm-1.in‚0 disk.dm-1.out‚ 20.58984375 disk.sda3.in‚0) proc.memory.percent‚18.04530620574951 network.recv‚285932' disk.nvme0c0n1.out‚11952.1962890625 disk.sda.out‚ 20.58984375 disk.sda1.in‚0# proc.memory.rssMB‚ 2956.54296875 cpu‚25.424184871765572 proc.cpu.threads‚17 network.sent‚575106) disk./.usagePercent‚78.99333301083212$ disk./.usageGB‚1177.057746887207 disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚11952.1962890625 disk.sda.in‚0 disk.sda1.out‚0" disk.dm-0.out‚11952.1962890625-]\§“Ö‚(:ˆ šÈëÔþ¶úÊ gpu.0.gpu‚11.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚33.0 gpu.0.powerWatts‚119.563' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9126 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®¯0j_ׂ(:Ô šÈëÔ“ÔÌ% disk./.usageGB‚1177.1639137268066% disk.nvme0c0n1.out‚12320.04296875 disk.dm-0.in‚ 1575.63671875 disk.dm-1.out‚21.28125' proc.memory.availableMB‚ 789.5859375 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2956.54296875) proc.memory.percent‚18.04530620574951 cpu‚25.419473834865475 proc.cpu.threads‚17) disk./.usagePercent‚79.00045795651755 disk.drbd1260.in‚0# disk.nvme0n1.out‚12320.04296875 disk.sda3.out‚21.28125 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.out‚12320.04296875$ memory_percent‚95.18074989318848 network.sent‚624310 network.recv‚290983 disk.sda2.in‚0 disk.drbd1260.out‚0.078125# disk.nvme0c0n1.in‚ 1575.63671875! disk.nvme0n1.in‚ 1575.63671875 disk.sda3.in‚0 disk.sda.out‚21.28125ÃWƒ‘Ø‚(:† ©ÈëÔ­©üÍ gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.735' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.947 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ÁøomÙ‚(:â ©ÈëÔ¸¦›Ï network.recv‚294778 disk.drbd1260.out‚ 0.08203125# disk.nvme0c0n1.in‚ 1575.63671875% disk.nvme0n1.out‚12369.1943359375$ memory_percent‚95.18074989318848# proc.memory.rssMB‚ 2956.54296875 proc.cpu.threads‚17 network.sent‚629564( disk./.usagePercent‚79.0004515563292$ disk./.usageGB‚1177.163818359375 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚12369.1943359375 disk.sda3.out‚ 21.69921875 disk.sda.out‚ 21.69921875 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 1575.63671875 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1575.63671875" disk.dm-0.out‚12369.1943359375 disk.dm-1.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 21.69921875' proc.memory.availableMB‚ 789.5859375) proc.memory.percent‚18.04530620574951 cpu‚25.41309012411358鸓ڂ(:ˆ ¸ÈëÔíÌšÌ gpu.0.gpu‚19.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.226' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.4452 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î­Ó«kÛ‚(:à ¸ÈëÔüö¼Í proc.cpu.threads‚17 network.recv‚299324$ disk./.usageGB‚1177.163818359375! disk.nvme0n1.in‚ 1575.63671875% disk.nvme0n1.out‚12608.4638671875 disk.sda3.in‚0 disk.dm-1.out‚ 22.3984375 disk.sda2.in‚0 disk.drbd1260.in‚0 network.sent‚635448( disk./.usagePercent‚79.0004515563292' disk.nvme0c0n1.out‚12608.4638671875 disk.sda3.out‚ 22.3984375 disk.sda.out‚ 22.3984375 disk.dm-0.in‚ 1575.63671875 disk.dm-1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1575.63671875 disk.sda.in‚0 disk.sda1.in‚0" disk.dm-0.out‚12608.4638671875 disk.drbd1260.out‚ 0.08203125' proc.memory.availableMB‚ 789.5703125) proc.memory.percent‚18.04530620574951 cpu‚25.406942550304336 disk.sda1.out‚0$ memory_percent‚95.18084526062012# proc.memory.rssMB‚ 2956.54296875Z½ò“Ü‚(:ˆ ÇÈëÔŸ…’Ì gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚34.0 gpu.0.powerWatts‚161.754' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u„ÕDdÝ‚(:Ù ÇÈëÔ’µÈÎ disk.sda3.out‚ 22.74609375 disk.dm-0.in‚ 1604.171875 network.recv‚303939 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 1604.171875 disk.dm-0.out‚12734.23046875$ memory_percent‚95.17934322357178) proc.memory.percent‚18.04530620574951 cpu‚25.401188062568735# disk.nvme0n1.out‚12734.23046875) disk./.usagePercent‚79.00079332638741 disk.dm-1.out‚ 22.74609375% disk.nvme0c0n1.out‚12734.23046875 disk.sda.in‚0 disk.sda.out‚ 22.74609375 disk.sda1.out‚0( proc.memory.availableMB‚ 789.81640625 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 1604.171875 disk.sda1.in‚0# proc.memory.rssMB‚ 2956.54296875 proc.cpu.threads‚17 network.sent‚643584% disk./.usageGB‚1177.1689109802246 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.0859375 disk.sda3.in‚0ïÏÞ‚(:’ ÖÈëÔÿ¦–Í gpu.0.gpu‚23.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.407302856445312* gpu.0.memoryAllocatedBytes‚ 14952759296 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.264799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÅPaß‚(:Ö ÖÈëÔ¸ïôÏ cpu‚25.395649972029602 network.sent‚648656) disk./.usagePercent‚78.99921964807443 disk.sda1.in‚0 disk.dm-0.in‚ 1730.39453125( proc.memory.availableMB‚ 789.81640625# proc.memory.rssMB‚ 2956.54296875 proc.cpu.threads‚17! disk.nvme0n1.in‚ 1730.39453125 disk.nvme0n1.out‚ 13160.59375 disk.sda.out‚ 23.23046875 disk.sda1.out‚0 disk.dm-0.out‚ 13159.71875 disk.dm-1.in‚0 disk.dm-1.out‚ 23.23046875$ memory_percent‚95.17934322357178 network.recv‚307708% disk./.usageGB‚1177.1454620361328 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1730.39453125) proc.memory.percent‚18.04530620574951 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0859375" disk.nvme0c0n1.out‚ 13160.59375 disk.sda3.in‚0 disk.sda3.out‚ 23.23046875 disk.sda.in‚06š”0à+ _wandbruntime‚697  _runtime‚697øOê†Äá¾ _runtime‚ 697.429062016 _step‚28 & lr-AdamW/pg1‚0.0005562544130835882 & lr-AdamW/pg2‚0.0005562544130835882 trainer/global_step‚399 % _timestamp‚1.7885359021178656e+097º—"Èâ _runtime‚ 697.429062016 _step‚28 & lr-AdamW/pg1‚0.0005562544130835882 & lr-AdamW/pg2‚0.0005562544130835882 trainer/global_step‚399 % _timestamp‚1.7885359021178656e+09™‡š/0ã+ _wandbruntime‚697  _runtime‚697R^@ Å'ä¿O OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2323787 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32671491131904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32671473128960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448332875264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8074794 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9949437 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7156837 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2323147 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8074794 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36111296157184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33119806004224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32076448372800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32671473128960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7156195 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚29 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 447558900016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449242079744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36111272217088 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 468916917248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 468681076736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31628889472784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18049077 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4833050 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31628872433424 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 446650932560 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8074792 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33119806004224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8074792 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32671491131904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18024229 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 448332875264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 trainer/global_step‚399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33120733211648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9974283 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33120733211648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4833048 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36579953293824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 _runtime‚ 697.430820881—éû´É'åÃO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33120733211648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9974283 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚33120733211648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14399045632 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4833048 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36579953293824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 _runtime‚ 697.430820881 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2323787 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚32671491131904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32671473128960 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 448332875264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8074794 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9949437 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7156837 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2323147 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8074794 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36111296157184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33119806004224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259780608 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚32076448372800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32671473128960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7156195 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚29 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1221 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 447558900016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 449242079744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36111272217088 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 468916917248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 468681076736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31628889472784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18049077 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4833050 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31628872433424 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 446650932560 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8074792 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33119806004224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8074792 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32671491131904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18024229 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 448332875264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 trainer/global_step‚399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615v,Vë0æ+ _wandbruntime‚698  _runtime‚698EW‰\“ çÉM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18069220 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885359035454097e+09 _runtime‚ 698.857131244 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18094094 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8094952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36671610128384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18094094 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2329571 =7DeviceStatsMonitor.on_train_batch_end/num_alll;U¼<oc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31707907067328 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 448676173502 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 450363553280 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4845113 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 447768205838 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32753092214784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36671367138304 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9974268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 450363553280 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8094954 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7174686 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36201521861632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4845115 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32156583240830 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18069220 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36201497921536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9999140 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚30 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7174079 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9974268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 trainer/global_step‚399 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33203473771008 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8094954 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9999140 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2328966 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 449454335488 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 470088266752 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32155658233806¹2ßžÓ&èÍM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚569 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8094952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32753110217728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33202546550272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18094094 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8094952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36671610128384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18094094 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2329571 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31707907067328 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 448676173502 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 450363553280 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚569 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4845113 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1221 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 447768205838 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32753092214784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36671367138304 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9974268 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 450363553280 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8094954 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7174686 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36201521861632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14399045632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4845115 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32156583240830 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚569 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18069220 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36201497921536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9999140 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚30 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7174079 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9974268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 trainer/global_step‚399 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33203473771008 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8094954 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9999140 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2328966 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 449454335488 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 470088266752 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32155658233806 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18069220 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885359035454097e+09 _runtime‚ 698.857131244 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2616732 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 504420300288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35579752160624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37257774927872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5436300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40622557439488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚449 _step‚33 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 504420300288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11194339 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40622581379584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9082794 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1226 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37256847720448 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37256847720448 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9082792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20301979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11219185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11194339 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 527339230208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 527560390656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35579769199984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8053030 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816ØM_ Í'ûÇO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36752445423104 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 505329504768 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41149896669696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37256847720448 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37256847720448 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9082792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20301979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14399045632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1193279488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14399045632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚569 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11219185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11194339 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 527339230208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 527560390656 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35579769199984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8053030 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41150141770240 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36083205355108 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36082280348292 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 503436155124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 505329504768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36752427420160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2617345 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1193279488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36752427420160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 502528187668 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5436298 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8053645 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚569 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36752445423104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221160448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9082792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2616732 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚569 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 504420300288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35579752160624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1193279488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37257774927872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5436300 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40622557439488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚449 _step‚33 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 504420300288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11194339 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40622581379584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9082794 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1226 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 505541760512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1226 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2622519 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 503645460946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9102954 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11244042 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 505541760512 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8071503 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235827200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1193279488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40712807084032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40712783143936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 506450978304 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9102954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41241277352960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37340515487232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20322122 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11244042 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36162415216114 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41241537120256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11219170 % _timestamp‚1.7885359831524348e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8070882 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5448363 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 trainer/global_step‚449 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9102952 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20346996 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9102952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36834046505984 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37339588266496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20346996 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35658786794528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35658769755168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 Û” þÑM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37340515487232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20322122 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 778.465282589 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1189 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37340515487232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20322122 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11244042 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36162415216114 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41241537120256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11219170 % _timestamp‚1.7885359831524348e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1193279488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8070882 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5448363 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14399045632 trainer/global_step‚449 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9102952 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20346996 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9102952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36834046505984 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37339588266496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20346996 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35658786794528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35658769755168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 505541760512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1226 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14399045632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2622519 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 503645460946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1184 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9102954 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11244042 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1193279488 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 505541760512 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8071503 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235827200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 SDDeviceStatsMonitor.on_train_batch_end/resbw4xÖerved_bytes.small_pool.peak‚ 1193279488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40712807084032 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40712783143936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 506450978304 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9102954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41241277352960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 558405442776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40833399714304 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 560507725312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 560507725312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6039548 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22530033 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41393889436672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40089962337416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10090794 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10090792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2911284 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40089037330600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40833399714304 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 561416929792 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22554881 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8950175 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39530648927184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚37 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45720205349888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39530631887824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2910627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6039550 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41394816644096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40833381711360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45133866601984 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 586338747904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 586113393152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40833381711360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚657 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45133842661888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 558405442776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚40833399714304 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 560507725312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 560507725312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚659 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6039548 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22530033 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41393889436672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40089962337416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10090794 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10090792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2911284 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40089037330600 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40833399714304 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 561416929792 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22554881 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8950175 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39530648927184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚37 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45720205349888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39530631887824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2910627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6039550 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41394816644096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40833381711360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45133866601984 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 586338747904 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 586113393152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40833381711360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚657 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45133842661888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22599898 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41476629982720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6ãSËNÀ#051615 % _timestamp‚1.7885360555492828e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41477557203456 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2916552 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40170097205446 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10110952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10110954 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22599898 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8968165 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45811609882624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12464072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10110954 _step‚38 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 560430683718 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12488944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40915000797184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6051613 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12488944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40915000797184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561629185536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22575024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 562538403328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41477557203456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45811366892544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39609649482368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45224092306432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39609666521728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12464072 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40915018800128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587298526208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22599898 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8968165 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45811609882624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12464072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10110954 _step‚38 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 560430683718 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12488944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40915000797184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6051613 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12488944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40915000797184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 561629185536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22575024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 562538403328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41477557203456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45811366892544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39609649482368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45224092306432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39609666521728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12464072 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40915018800128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 587298526208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22599898 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41476629982720 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6051615 % _timestamp‚1.7885360555492828e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41477557203456 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2916552 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40170097205446 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10110952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615&àl€²0•+ _wandbruntime‚850  _runtime‚850&CB–¼ /loss/FRt1/detection_loss‚3.6234151899814604 1loss/SRqqt3/detection_loss‚1.7552057020849996 0loss/SRqqt4/detection_loss‚0.945812624087122 .loss/FBt4/detection_loss‚9.823509603738785 _step‚39 0loss/FRt3/assignment_loss‚5.1456005573272705 /loss/FBt4/assignment_loss‚0.576646164059639 /loss/FRt2/detection_loss‚2.8426084692989075 %loss/total_loss‚74.35524766014798 epoch‚0 % _timestamp‚1.7885360555571249e+09 /loss/FRt1/assignment_loss‚6.379331922531128 /loss/FRt4/assignment_loss‚6.325785542789259 _runtime‚ 850.868045254 loss/FBt3/assignment_loss‚0 0loss/SRqqt2/detection_loss‚2.609688714146614 .loss/FBt1/detection_loss‚3.360818530122439 /loss/FBt2/detection_loss‚1.5519408271980606 /loss/FRt4/detection_loss‚1.7163653828602026 /loss/FRt2/assignment_loss‚6.470448453085763 2loss/SRqqt2/assignment_loss‚3.1607372045516966 2loss/SRqqt3/assignment_loss‚2.8861772503171648 0loss/FBt2/assignment_loss‚0.3163102743634812 .loss/FBt3/detection_loss‚2.062788486480713 2loss/SRqqt1/assignment_loss‚3.5431629717350006 2loss/SRqqt4/assignment_loss‚2.8074575595766587 0loss/FBt1/assignment_loss‚0.7118339582035939 /loss/FRt3/detection_loss‚2.3188039289330535 1loss/SRqqt1/detection_loss‚3.4207983426749706 trainer/global_step‚499°ÒÛÙF—À _runtime‚ 850.868045254 loss/FBt3/assignment_loss‚0 0loss/SRqqt2/detection_loss‚2.609688714146614 .loss/FBt1/detection_loss‚3.360818530122439 /loss/FBt2/detection_loss‚1.5519408271980606 /loss/FRt4/detection_loss‚1.7163653828602026 /loss/FRt2/assignment_loss‚6.470448453085763 2loss/SRqqt2/assignment_loss‚3.1607372045516966 2loss/SRqqt3/assignment_loss‚2.8861772503171648 0loss/FBt2/assignment_loss‚0.3163102743634812 .loss/FBt3/detection_loss‚2.062788486480713 2loss/SRqqt1/assignment_loss‚3.5431629717350006 2loss/SRqqt4/assignment_loss‚2.8074575595766587 0loss/FBt1/assignment_loss‚0.7118339582035939 /loss/FRt3/detection_loss‚2.3188039289330535 1loss/SRqqt1/detection_loss‚3.4207983426749706 trainer/global_step‚499 /loss/FRt1/detection_loss‚3.6234151899814604 1loss/SRqqt3/detection_loss‚1.7552057020849996 0loss/SRqqt4/detection_loss‚0.945812624087122 .loss/FBt4/detection_loss‚9.823509603738785 _step‚39 0loss/FRt3/assignment_loss‚5.1456005573272705 /loss/FBt4/assignment_loss‚0.576646164059639 /loss/FRt2/detection_loss‚2.8426084692989075 %loss/total_loss‚74.35524766014798 epoch‚0 % _timestamp‚1.7885360555571249e+09 /loss/FRt1/assignment_loss‚6.379331922531128 /loss/FRt4/assignment_loss‚6.325785542789259'TíÆ‹o˜‚(:ä ûÉëÔ©±ÚÎ disk.sda3.out‚ 28.609375 disk.dm-1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 789.09765625 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚809297% disk./.usageGB‚1177.7678298950195& disk.nvme0n1.out‚16403.43896484375 disk.sda.out‚ 28.609375 disk.dm-0.in‚ 2031.30078125# proc.memory.rssMB‚ 2956.55859375) disk./.usagePercent‚79.04098727728574 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 28.609375 disk.drbd1260.in‚0$ memory_percent‚95.18373012542725* proc.memory.percent‚18.045401573181152 network.recv‚355554# disk.nvme0c0n1.in‚ 2031.30078125( disk.nvme0c0n1.out‚16403.43896484375 disk.sda3.in‚0# disk.dm-0.out‚16403.43896484375 disk.sda2.in‚0 disk.drbd1260.out‚ 0.11328125 cpu‚25.348088734680076! disk.nvme0n1.in‚ 2031.30078125ò¢Ô“™‚(:ˆ ûÉëÔ¸œªÑ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚153.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ØÌ“š‚(:ˆ ŠÊëÔä‚õÊ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.732' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ye¦q›‚(:æ ŠÊëÔÒ„³Í disk.sda3.out‚ 29.34765625 disk.sda.out‚ 29.34765625# disk.dm-0.out‚16794.76708984375 proc.cpu.threads‚17 network.sent‚845000 network.recv‚360426 disk.sda2.out‚0 disk.drbd1260.out‚ 0.11328125 disk.sda.in‚0# proc.memory.rssMB‚ 2956.55859375* proc.memory.percent‚18.045401573181152% disk./.usageGB‚1177.7678298950195 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 2218.3046875& disk.nvme0n1.out‚16794.76708984375 disk.sda1.out‚0 disk.dm-1.out‚ 29.34765625' proc.memory.availableMB‚ 789.0859375 cpu‚25.344382602323243 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 2218.3046875( disk.nvme0c0n1.out‚16794.76708984375 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 2218.3046875 disk.dm-1.in‚0$ memory_percent‚95.18380165100098) disk./.usagePercent‚79.04098727728574 ‘zÀqœ‚(:æ ™ÊëÔÂæüÌ disk.drbd1260.out‚ 0.1171875 disk.nvme0n1.in‚ 2218.3046875& disk.nvme0n1.out‚17032.39208984375 proc.cpu.threads‚17 disk.dm-0.in‚ 2218.3046875 disk.dm-1.out‚ 29.73828125" disk.nvme0c0n1.in‚ 2218.3046875 disk.sda.in‚0 disk.sda.out‚ 29.73828125* proc.memory.percent‚18.045401573181152 cpu‚25.340746859833875 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.18382549285889( proc.memory.availableMB‚ 789.08203125# proc.memory.rssMB‚ 2956.55859375 network.sent‚852029 network.recv‚363784# disk.dm-0.out‚17032.39208984375 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚17032.39208984375 disk.sda3.out‚ 29.73828125) disk./.usagePercent‚79.04256121160624% disk./.usageGB‚1177.7912826538086 disk.dm-1.in‚0©íÔë“‚(:ˆ ™ÊëÔïÔ„Í gpu.0.gpu‚42.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.284' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oó“ž‚(:ˆ ¨ÊëÔƒù˜Ê gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É©^qŸ‚(:æ ¨ÊëÔÊ«äÍ disk.drbd1260.out‚ 0.1171875 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚17181.04833984375 disk.dm-1.in‚0 cpu‚25.337158547480744 disk.sda.in‚0 disk.dm-0.in‚ 2218.3046875$ memory_percent‚95.18840312957764 proc.cpu.threads‚17 network.recv‚368285% disk./.usageGB‚1177.7912826538086 disk.sda2.out‚0 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 2218.3046875( disk.nvme0c0n1.out‚17181.04833984375 disk.nvme0n1.in‚ 2218.3046875& disk.nvme0n1.out‚17181.04833984375 network.sent‚856073) disk./.usagePercent‚79.04256121160624 disk.sda3.out‚ 30.11328125 disk.sda.out‚ 30.11328125 disk.dm-1.out‚ 30.11328125( proc.memory.availableMB‚ 788.33203125# proc.memory.rssMB‚ 2956.55859375* proc.memory.percent‚18.045401573181152 disk.sda2.in‚0|…b ‚(:× ·ÊëÔéÓ†Î* proc.memory.percent‚18.045401573181152 cpu‚25.33396938375009 proc.cpu.threads‚17 network.recv‚373020 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 2218.3046875# proc.memory.rssMB‚ 2956.55859375 network.sent‚862256% disk./.usageGB‚1177.7443466186523 disk.sda1.out‚0 disk.dm-0.in‚ 2218.3046875" disk.dm-0.out‚17430.8466796875' disk.nvme0c0n1.out‚17430.8466796875 disk.nvme0n1.in‚ 2218.3046875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚30.8125 disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚17430.8466796875 disk.sda3.out‚30.8125 disk.sda.in‚0( proc.memory.availableMB‚ 789.08203125) disk./.usagePercent‚79.03941129490495 disk.sda1.in‚0 disk.drbd1260.out‚ 0.12109375 disk.sda.out‚30.8125$ memory_percent‚95.18382549285889lX,¡‚(:’ ·ÊëÔ»®·Í gpu.0.gpu‚8.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–È„0¢+ _wandbruntime‚921  _runtime‚921®aÄ£¾ _runtime‚ 921.010506323 _step‚40 & lr-AdamW/pg1‚0.0007653726134909523 & lr-AdamW/pg2‚0.0007653726134909523 trainer/global_step‚549 % _timestamp‚1.7885361256994152e+09 ¿SȤ _runtime‚ 921.010506323 _step‚40 & lr-AdamW/pg1‚0.0007653726134909523 & lr-AdamW/pg2‚0.0007653726134909523 trainer/global_step‚549 % _timestamp‚1.7885361256994152e+09(ìþA0¥+ _wandbruntime‚921  _runtime‚921 3dF(¦ÇO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44914354005504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 617504354816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3206947 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49645127884288 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13684143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6642800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_th“¢µ¥rain_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45531858360320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50290213329920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 645085445632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44914336002560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚41 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44914336002560 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3206282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614282697884 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45531858360320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43481528654384 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚ 921.012714315 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11098794 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6642798 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 616595150336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13708989 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43481511615024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 645327577600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24807783 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45530931152896 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9849080 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24782935 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9849747 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 266072064 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44096719319724 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11098792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11098792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49645151824384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44095794312908 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 % _timestamp‚1.7885361257013862e+09’: fÑ'§ËO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44914354005504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 617504354816 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3206947 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49645127884288 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13684143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6642800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45531858360320 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50290213329920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 645085445632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44914336002560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚41 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44914336002560 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3206282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 614282697884 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45531858360320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43481528654384 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚ 921.012714315 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11098794 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6642798 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 616595150336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13708989 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43481511615024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 645327577600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24807783 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45530931152896 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9849080 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24782935 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9849747 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 266072064 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44096719319724 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11098792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11098792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49645151824384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44095794312908 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 % _timestamp‚1.7885361257013862e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944)å^‘0¨+ _wandbruntime‚922  _runtime‚922dê46×&©ÑM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11118954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 _runtime‚ 922.340625093 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 618625828352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 617716610560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49735377528832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚42 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43560546248928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13708974 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50381602705920 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44995973091328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44995955088384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44995955088384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24827926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24827926 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45614598919680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 646478652928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9867711 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44175929180730 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43560529209568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11118952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 618625828352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11118952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13708974 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44176854187754 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 616307938826 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24852800 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 617716610560 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9867050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45614598919680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44995973091328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6654865 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24852800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3212846yP»`ªÕM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24827926 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45614598919680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚661 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253475840 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 646478652928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9867711 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44175929180730 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43560529209568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11118952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 618625828352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11118952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13708974 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44176854187754 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 616307938826 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24852800 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 617716610560 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9867050 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45614598919680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44995973091328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6654865 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚659 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24852800 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3212846 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11118954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pooDµ¦\{l.peak‚ 184953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 _runtime‚ 922.340625093 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 618625828352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 617716610560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49735377528832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚42 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43560546248928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13708974 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50381602705920 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44995973091328 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44995955088384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44995955088384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229535744 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24827926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0*JC*0«+ _wandbruntime‚922  _runtime‚922~ ýZU¬Ï _step‚43 /loss/FRt3/assignment_loss‚5.502399748944222 /loss/FRt4/assignment_loss‚5.513230670066107 2loss/SRqqt3/assignment_loss‚2.6565004273464807 1loss/SRqqt1/detection_loss‚3.5128624255840597 trainer/global_step‚549 2loss/SRqqt4/assignment_loss‚2.7247738335646834 0loss/FBt1/assignment_loss‚0.6299927407716706 /loss/FRt2/detection_loss‚3.7232431396842003 %loss/total_loss‚69.88913443002905 % _timestamp‚1.7885361270317116e+09 _runtime‚ 922.342661321 /loss/FRt1/assignment_loss‚6.168222069740295 1loss/SRqqt4/detection_loss‚0.9980698948835626 /loss/FBt3/detection_loss‚5.0558459758758545 epoch‚0 2loss/FBt3/assignment_loss‚0.018456390593200922 0loss/SRqqt3/detection_loss‚1.768844154516333 /loss/FRt2/assignment_loss‚7.049764573574066 1loss/SRqqt1/assignment_loss‚2.806484506680415 1loss/SRqqt2/assignment_loss‚2.504205067952474 /loss/FBt4/assignment_loss‚0.966177485444913 1loss/SRqqt2/detection_loss‚3.3164196610450745 .loss/FBt1/detection_loss‚3.054845479818491 /loss/FBt2/detection_loss‚0.7589754158229146 /loss/FBt4/detection_loss‚3.3859106933369354 1loss/FBt2/assignment_loss‚0.32749771108484943 /loss/FRt1/detection_loss‚3.3402713196618214 .loss/FRt3/detection_loss‚2.316875897823496 /loss/FRt4/detection_loss‚1.7892651462129188{ÙîY­Ó %loss/total_loss‚69.88913443002905 % _timestamp‚1.7885361270317116e+09 _runtime‚ 922.342661321 /loss/FRt1/assignment_loss‚6.168222069740295 1loss/SRqqt4/detection_loss‚0.9980698948835626 /loss/FBt3/detection_loss‚5.0558459758758545 epoch‚0 2loss/FBt3/assignment_loss‚0.018456390593200922 0loss/SRqqt3/detection_loss‚1.768844154516333 /loss/FRt2/assignment_loss‚7.049764573574066 1loss/SRqqt1/assignment_loss‚2.806484506680415 1loss/SRqqt2/assignment_loss‚2.504205067952474 /loss/FBt4/assignment_loss‚0.966177485444913 1loss/SRqqt2/detection_loss‚3.3164196610450745 .loss/FBt1/detection_loss‚3.054845479818491 /loss/FBt2/detection_loss‚0.7589754158229146 /loss/FBt4/detection_loss‚3.3859106933369354 1loss/FBt2/assignment_loss‚0.32749771108484943 /loss/FRt1/detection_loss‚3.3402713196618214 .loss/FRt3/detection_loss‚2.316875897823496 /loss/FRt4/detection_loss‚1.7892651462129188 _step‚43 /loss/FRt3/assignment_loss‚5.502399748944222 /loss/FRt4/assignment_loss‚5.513230670066107 2loss/SRqqt3/assignment_loss‚2.6565004273464807 1loss/SRqqt1/detection_loss‚3.5128624255840597 trainer/global_step‚549 2loss/SRqqt4/assignment_loss‚2.7247738335646834 0loss/FBt1/assignment_loss‚0.6299927407716706 /loss/FRt2/detection_loss‚3.7232431396842003+ :SV“®‚(:ˆ ÆÊëÔ­ÉÌ gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÁn¯‚(:ã ÆÊëÔÜ ÖÐ disk.drbd1260.out‚ 0.12109375' disk.nvme0c0n1.out‚17806.0849609375! disk.nvme0n1.in‚ 2260.12109375 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 31.4140625$ memory_percent‚95.18599510192871% disk.nvme0n1.out‚17806.0849609375 disk.sda3.in‚0 network.sent‚866334# disk.nvme0c0n1.in‚ 2260.12109375 disk.sda.in‚0 disk.sda.out‚ 31.4140625 disk.dm-0.in‚ 2260.12109375 proc.cpu.threads‚17 network.recv‚375614 disk.sda2.out‚0 disk.sda1.out‚0" disk.dm-0.out‚17806.0849609375' proc.memory.availableMB‚ 788.7265625# proc.memory.rssMB‚ 2956.55859375* proc.memory.percent‚18.045401573181152 cpu‚25.330751668421026 disk.sda3.out‚ 31.4140625) disk./.usagePercent‚79.04098753329328% disk./.usageGB‚1177.7678337097168 disk.drbd1260.in‚03uË’°‚(:‡ ÕÊëÔ÷ƒºÌ gpu.0.gpu‚2.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚164.159' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8318 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€o‚Þh±‚(:Ý ÕÊëÔœ¢ÅÐ disk.drbd1260.out‚0.125 cpu‚25.32752592532913% disk./.usageGB‚1177.7678146362305! disk.nvme0n1.in‚ 2398.27734375 disk.sda3.in‚0 disk.sda3.out‚ 31.8359375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 network.sent‚915186% disk.nvme0n1.out‚18213.6748046875 disk.dm-0.in‚ 2398.27734375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 2398.27734375' disk.nvme0c0n1.out‚18213.6748046875$ memory_percent‚95.18601894378662( proc.memory.availableMB‚ 788.72265625( disk./.usagePercent‚79.0409862532556 disk.sda.in‚0" disk.dm-0.out‚18213.6748046875# proc.memory.rssMB‚ 2956.55859375* proc.memory.percent‚18.045401573181152 proc.cpu.threads‚17 network.recv‚380118 disk.sda.out‚ 31.8359375 disk.sda1.out‚0 disk.dm-1.out‚ 31.8359375ðwÜ“²‚(:ˆ äÊëÔÑèÙÍ gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|¸ã°i³‚(:Þ äÊëÔÊê÷Ï! disk.nvme0n1.in‚ 2398.27734375 disk.sda3.out‚ 32.43359375% proc.memory.availableMB‚ 788.96875# proc.memory.rssMB‚ 2956.55859375 cpu‚25.324670170383353 network.sent‚921026) disk./.usagePercent‚79.04257503601309$ disk./.usageGB‚1177.791488647461 disk.dm-0.in‚ 2398.27734375" disk.dm-0.out‚18276.3388671875 disk.dm-1.out‚ 32.43359375 disk.drbd1260.out‚0.125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚18276.3388671875$ memory_percent‚95.18451690673828* proc.memory.percent‚18.045401573181152 proc.cpu.threads‚17 network.recv‚384590 disk.sda.out‚ 32.43359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚18276.3388671875# disk.nvme0c0n1.in‚ 2398.27734375µ(°ý´‚(:’ óÊëÔÍÝÉÌ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.71' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.541999999999998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q2jµ‚(:ß óÊëÔ­àŠÑ disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.12890625# disk.nvme0c0n1.in‚ 2398.27734375 disk.sda3.in‚0 disk.sda.out‚ 32.890625 disk.sda1.in‚0 network.recv‚389200% disk./.usageGB‚1177.7914695739746 disk.sda1.out‚0$ memory_percent‚95.18611431121826( proc.memory.availableMB‚ 788.70703125 proc.memory.rssMB‚ 2956.5625* proc.memory.percent‚18.045425415039062 proc.cpu.threads‚17( disk.nvme0c0n1.out‚18770.44189453125) disk./.usagePercent‚79.04257375597543# disk.dm-0.out‚18770.44189453125 disk.dm-1.in‚0! disk.nvme0n1.in‚ 2398.27734375& disk.nvme0n1.out‚18770.44189453125 cpu‚25.32156239980076 network.sent‚927237 disk.dm-1.out‚ 32.890625 disk.drbd1260.in‚0 disk.sda3.out‚ 32.890625 disk.sda.in‚0 disk.dm-0.in‚ 2398.277343758‡ò“¶‚(:ˆ ‚ËëÔºô¾Ë gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚120.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ_n·‚(:ã ‚ËëÔ¶ÍúÌ network.recv‚392656 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.12890625( disk.nvme0c0n1.out‚18770.84033203125! disk.nvme0n1.in‚ 2398.27734375 disk.sda.in‚0% disk./.usageGB‚1177.7944374084473 disk.sda3.in‚0 disk.sda3.out‚ 33.328125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.18616199493408 cpu‚25.31856997655344) disk./.usagePercent‚79.04277292983706# disk.nvme0c0n1.in‚ 2398.27734375 disk.sda.out‚ 33.328125 disk.dm-0.in‚ 2398.27734375 disk.dm-1.out‚ 33.328125# proc.memory.rssMB‚ 2956.56640625 disk.drbd1260.in‚0& disk.nvme0n1.out‚18770.84033203125# disk.dm-0.out‚18770.84033203125 disk.dm-1.in‚0( proc.memory.availableMB‚ 788.69921875* proc.memory.percent‚18.045449256896973 proc.cpu.threads‚17 network.sent‚932460Pbs[0¸+ _wandbruntime‚991  _runtime‚991¬„þ¹¼ trainer/global_step‚599 % _timestamp‚1.7885361963400562e+09 _runtime‚ 991.651544489 _step‚44 % lr-AdamW/pg1‚0.000835078680293407 % lr-AdamW/pg2‚0.000835078680293407ý(BRƺÀ % lr-AdamW/pg1‚0.000835078680293407 % lr-AdamW/pg2‚0.000835078680293407 trainer/global_step‚599 % _timestamp‚1.7885361963400562e+09 _runtime‚ 991.651544489 _step‚44,ÿøà0»+ _wandbruntime‚991  _runtime‚9914Ë—<Ï'¼ÉO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12106794 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12106792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 703920954368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3501137 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚599 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7246050 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚45 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10747829 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7246048 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54860334061056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14953891 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47432408381584 % _timestamp‚1.7885361963425257e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54860579161600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54156437046784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 671067920448 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49667972869120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54156413106688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784dÓe#½ÍO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚ 991.654030265 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 672682575360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 670159952992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49667972869120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54156413106688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12106794 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12106792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 703920954368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3501137 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 trainer/global_step‚599 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7246050 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚45 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10747829 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7246048 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54860334061056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14953891 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47432408381584 % _timestamp‚1.7885361963425257e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54860579161600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54156437046784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 671067920448 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚46 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12126952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49751640635904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 673804035584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14978748 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49751640635904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27080828 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10765073 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49076927382528 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 674713253376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49076909379584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3506960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14978748 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7258113 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27105702 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14953876 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27080828 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54246638811136 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 705328375808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49076927382528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14953876 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49750713415168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47511425976128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10765712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27105702 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12126952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚ 992.873858805 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 673804035584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 705111422976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54951991127040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54246662751232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0´ŽBöÝ&À×M REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 673804035584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 705111422976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54951991127040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54246662751232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚46 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12126952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49751640635904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 673804035584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14978748 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49751640635904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216952832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27080828 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10765073 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49076927382528 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 674713253376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49076909379584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3506960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14978748 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7258113 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27105702 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14953876 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27080828 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54246638811136 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 705328375808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚49076927382528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14953876 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49750713415168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47511425976128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10765712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27105702 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12126952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚ 992.873858805.綇0Á+ _wandbruntime‚992  _runtime‚992ÓØ”UÂÏ 1loss/SRqqt1/detection_loss‚2.9145149191220603 _step‚47 /loss/FRt2/detection_loss‚2.8764782094955446 /loss/FBt3/detection_loss‚3.7783093452453613 1loss/SRqqt1/assignment_loss‚3.155625573794047 0loss/FBt1/assignment_loss‚0.7004434956819751 _runtime‚ 992.876098356 % _timestamp‚1.7885361975652077e+09 2loss/SRqqt2/assignment_loss‚3.0342881441116334 0loss/FBt2/assignment_loss‚0.2737974342537422 1loss/FBt4/assignment_loss‚0.36372323088580744 /loss/FBt2/detection_loss‚0.9817958271079906 2loss/SRqqt3/assignment_loss‚2.6752477370459458 1loss/FBt3/assignment_loss‚0.19211509823799133 1loss/SRqqt4/detection_loss‚1.0524687878787518 %loss/total_loss‚67.19322044529056 epoch‚0 1loss/SRqqt4/assignment_loss‚2.668203646938006 1loss/SRqqt2/detection_loss‚3.3254599034786225 1loss/SRqqt3/detection_loss‚2.0043599597852806 trainer/global_step‚599 /loss/FRt3/assignment_loss‚5.863965897333054 /loss/FRt1/detection_loss‚2.9625657685101032 /loss/FRt3/detection_loss‚2.7115742024921237 /loss/FRt4/detection_loss‚1.6743208048453673 .loss/FBt1/detection_loss‚2.549837052822113 -loss/FBt4/detection_loss‚3.41538290977478 /loss/FRt1/assignment_loss‚5.812956273555756 /loss/FRt2/assignment_loss‚7.259642009735107 /loss/FRt4/assignment_loss‚4.946144213159401¬Û•\YÃÓ 1loss/SRqqt4/assignment_loss‚2.668203646938006 1loss/SRqqt2/detection_loss‚3.3254599034786225 1loss/SRqqt3/detection_loss‚2.0043599597852806 trainer/global_step‚599 /loss/FRt3/assignment_loss‚5.863965897333054 /loss/FRt1/detection_loss‚2.9625657685101032 /loss/FRt3/detection_loss‚2.7115742024921237 /loss/FRt4/detection_loss‚1.6743208048453673 .loss/FBt1/detection_loss‚2.549837052822113 -loss/FBt4/detection_loss‚3.41538290977478 /loss/FRt1/assignment_loss‚5.812956273555756 /loss/FRt2/assignment_loss‚7.259642009735107 /loss/FRt4/assignment_loss‚4.946144213159401 1loss/SRqqt1/detection_loss‚2.9145149191220603 _step‚47 /loss/FRt2/detection_loss‚2.8764782094955446 /loss/FBt3/detection_loss‚3.7783093452453613 1loss/SRqqt1/assignment_loss‚3.155625573794047 0loss/FBt1/assignment_loss‚0.7004434956819751 _runtime‚ 992.876098356 % _timestamp‚1.7885361975652077e+09 2loss/SRqqt2/assignment_loss‚3.0342881441116334 0loss/FBt2/assignment_loss‚0.2737974342537422 1loss/FBt4/assignment_loss‚0.36372323088580744 /loss/FBt2/detection_loss‚0.9817958271079906 2loss/SRqqt3/assignment_loss‚2.6752477370459458 1loss/FBt3/assignment_loss‚0.19211509823799133 1loss/SRqqt4/detection_loss‚1.0524687878787518 %loss/total_loss‚67.19322044529056 epoch‚0/³;©rÄ‚(:ç ‘ËëÔÉéšÏ# disk.nvme0c0n1.in‚ 2563.58984375 disk.sda3.in‚0( proc.memory.availableMB‚ 788.89453125 proc.cpu.threads‚17 disk.sda3.out‚ 33.76171875 network.sent‚937586) disk./.usagePercent‚79.04139356124257# disk./.usageGB‚1177.77388381958 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1328125! disk.nvme0n1.in‚ 2563.58984375 disk.sda.out‚ 33.76171875 disk.sda2.in‚0& disk.nvme0n1.out‚19247.87841796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 2563.58984375 disk.dm-1.out‚ 33.76171875# proc.memory.rssMB‚ 2956.56640625* proc.memory.percent‚18.045449256896973 disk.sda2.out‚0( disk.nvme0c0n1.out‚19247.87841796875 disk.sda.in‚0# disk.dm-0.out‚19247.87841796875 disk.dm-1.in‚0$ memory_percent‚95.18496990203857 cpu‚25.316532513817307 network.recv‚396413t.X’žÅ‚(:“ ‘ËëÔÐô£Î gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.591' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.918200000000002 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’9#B“Æ‚(:ˆ  ËëÔó¬îË gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚121.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»¸C·oÇ‚(:ä  ËëÔŸ™¿Ï( disk.nvme0c0n1.out‚19478.84326171875 network.sent‚987134 network.recv‚402002 disk.drbd1260.in‚0# disk.dm-0.out‚19478.84326171875 disk.dm-1.out‚ 34.3984375 proc.cpu.threads‚17 disk.sda2.in‚0& disk.nvme0n1.out‚19478.84326171875 disk.sda3.in‚0 disk.dm-0.in‚ 2563.58984375$ memory_percent‚95.18914222717285' proc.memory.availableMB‚ 788.2109375# proc.memory.rssMB‚ 2956.56640625* proc.memory.percent‚18.045449256896973) disk./.usagePercent‚79.04137103257955% disk./.usageGB‚1177.7735481262207 disk.sda2.out‚0 disk.drbd1260.out‚ 0.1328125# disk.nvme0c0n1.in‚ 2563.58984375 disk.sda.out‚ 34.3984375 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 2563.58984375 disk.sda3.out‚ 34.3984375 disk.sda.in‚0 disk.dm-1.in‚0 cpu‚25.31371222685722ŽÞÉZ“È‚(:ˆ ¯ËëÔÞ¯ÆÌ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.838' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t±e uÉ‚(:ê ¯ËëÔÉ °Ï disk.drbd1260.out‚ 0.13671875( disk.nvme0c0n1.out‚19801.66455078125 disk.sda3.in‚0 cpu‚25.311449498145702 network.sent‚993332 network.recv‚406464! disk.nvme0n1.in‚ 2563.58984375 disk.sda3.out‚ 34.87890625 disk.sda.in‚0 disk.sda.out‚ 34.87890625# disk.dm-0.out‚19801.66455078125* proc.memory.percent‚18.045449256896973& disk.nvme0n1.out‚19801.66455078125 disk.dm-1.out‚ 34.87890625$ memory_percent‚95.18616199493408( proc.memory.availableMB‚ 788.69921875 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 2563.58984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 2563.58984375 disk.dm-1.in‚0# proc.memory.rssMB‚ 2956.56640625 proc.cpu.threads‚17) disk./.usagePercent‚79.04294547891514% disk./.usageGB‚1177.7970085144043 disk.sda2.in‚0 disk.sda2.out‚0¨‚Ã"“Ê‚(:ˆ ¾ËëÔ”ä¤Î gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ØitË‚(:é ¾ËëÔú”ÄÓ disk.dm-0.in‚ 2563.58984375' proc.memory.availableMB‚ 788.6953125# proc.memory.rssMB‚ 2956.56640625 cpu‚25.308589059313135 network.recv‚409892) disk./.usagePercent‚79.04294727096787& disk.nvme0n1.out‚20142.53173828125 disk.sda3.in‚0$ memory_percent‚95.18618583679199* proc.memory.percent‚18.045449256896973 network.sent‚998504% disk./.usageGB‚1177.7970352172852 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.13671875! disk.nvme0n1.in‚ 2563.58984375 disk.sda1.in‚0# disk.dm-0.out‚20142.53173828125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 2563.58984375( disk.nvme0c0n1.out‚20142.53173828125 disk.sda3.out‚ 35.23828125 disk.sda.out‚ 35.23828125 disk.sda1.out‚0 disk.dm-1.out‚ 35.23828125 disk.sda.in‚0ÃCVÿ2Ì- _wandbruntime‚1063  _runtime‚1063oTƧÅÍ¿ & lr-AdamW/pg1‚0.0009047847470958616 & lr-AdamW/pg2‚0.0009047847470958616 trainer/global_step‚649 % _timestamp‚1.7885362686255722e+09 _runtime‚1063.936587623 _step‚48·T@ÉÎà trainer/global_step‚649 % _timestamp‚1.7885362686255722e+09 _runtime‚1063.936587623 _step‚48 & lr-AdamW/pg1‚0.0009047847470958616 & lr-AdamW/pg2‚0.00090478474709586160·4ϵ2Ï- _wandbruntime‚1063  _runtime‚1063ƒ‘ / ÐÉO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53805014585344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 726945175556 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7849298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58667698329088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788536268626785e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728770000384 SDDeviceStatsMonitor.o)ã+ n_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51383271069424 _step‚49 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16198793 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53076262587904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29288739 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13114792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13114792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58667722269184 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 762863173632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1063.937881544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11646348 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53805941792768 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52109308277524 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53076244584960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728770000384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13114794 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59430561502720·}úÓ'ÑÍO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16198793 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3797048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53805014585344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53076244584960 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 729679204864 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 763092722688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚649 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 729679204864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229549056 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13114794 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59430561502720 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53805014585344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1253 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 726945175556 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7849298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58667698329088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788536268626785e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 728770000384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51383271069424 _step‚49 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16198793 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53076262587904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253489152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29288739 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13114792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13114792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58667722269184 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 762863173632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚1063.937881544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11646348 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53805941792768 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52109308277524 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53076244584960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728770000384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53157881673728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59522216033280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51462288663968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13134952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58757947973632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52189443145346 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885362697241707e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 730800678400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13134954 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52190368152370 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚1065.035830463 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13134952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29333730 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58757924033536 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16223650 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16223650 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29358604 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59521968848896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53888682352128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29358604 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53157863670784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53157881673728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11663595 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7861365 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 764044815360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3802890 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3802232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729891460608 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 764268059648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51462305703328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53157881673728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59522216033280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51462288663968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1253 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/aSýJº¡ctive_bytes.all.current‚ 927220736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13134952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58757947973632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚52189443145346 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885362697241707e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 730800678400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13134954 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚52190368152370 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚1065.035830463 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13134952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29333730 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58757924033536 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16223650 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16223650 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29358604 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59521968848896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53888682352128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29358604 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53157863670784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53157881673728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11663595 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7861365 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 764044815360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3802890 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3802232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729891460608 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 764268059648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51462305703328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8452550 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57157216879104 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 784857425408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4092273 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57157216879104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚699 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 785766629888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17418849 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17443695 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14122792 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56116065259832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57157198876160 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 785766629888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8452548 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63179007491584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12544821 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55334167835984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12545497 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55334150796624 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57942983508992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 $ _timestamp‚1.788536340952025e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14122794 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57157198876160 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31566489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 821927683584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17418849 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 821708620288 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000935175168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56116990266648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57942056301568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782822430664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57942056301568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚1136.263183316 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 781914463208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784ª]³ä çÍO ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 785766629888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17418849 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17443695 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14122792 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56116065259832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57157198876160 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 785766629888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8452548 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63179007491584 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12544821 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55334167835984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12545497 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55334150796624 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57942983508992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 $ _timestamp‚1.788536340952025e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14122794 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57157198876160 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31566489 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 821927683584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17418849 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 821708620288 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000935175168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚56116990266648 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57942056301568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782822430664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57942056301568 XJDeviceStatsMonitor.on_train_batch_start/alloc!"Eh·ated_bytes.small_pool.current‚ 909204480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚1136.263183316 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 781914463208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64000692171776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8452550 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57157216879104 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 784857425408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4092273 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57157216879104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚6995 •ê2è- _wandbruntime‚1137  _runtime‚1137áÍÃ[Ú&éÔM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57238835964928 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57238817961984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14142952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17468552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17443680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 823106528768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31586632 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 783031736486 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚699 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 785978885632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57238835964928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _runtime‚1137.478512968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14142952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 _step‚54 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64092339724800 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56196200127654 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12562855 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17468552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31611506 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12563522 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58024796847616 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58024796847616 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17468552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17443680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 823106528768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31586632 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 783031736486 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚699 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 785978885632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57238835964928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _runtime‚1137.478512968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14142952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 _step‚54 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64092339724800 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56196200127654 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12562855 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17468552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31611506 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12563522 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62079098017792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62080025225216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13445120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33819391 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4389320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 880954214912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚57 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9055800 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18688597 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 837791718316 _runtime‚1208.898230824 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61238153167360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61238171170304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59285030523824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚749 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62079098017792 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 838699685772 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4388639 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 880722568704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 841854054912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚61238153167360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1259 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61238171170304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59285030523824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚749 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62079098017792 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 838699685772 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4388639 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 880722568704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 841854054912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62079098017792 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62080025225216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13445120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33819391 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4389320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 880954214912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚57 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 9092044809chôu2þ- _wandbruntime‚1210  _runtime‚1210Ê÷œæÚ&ÿÔM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13462963 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 842975528448 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68662640656896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61319772253184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33839534 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18713454 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68662389278208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 227438592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15150954 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67780494478336 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60203882116986 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18713454 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13462297 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62162765784576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15150952 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 839816959258 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9067865 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61319790256128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚1210.344924664 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62162765784576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 251378688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33864408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61319790256128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67780518418432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 842975528448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 838908991594 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33864408 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18688582 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144‚½‡= €ØM `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 882122238464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13462297 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62162765784576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15150952 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 839816959258 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9067865 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61319790256128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚1210.344924664 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62162765784576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 251378688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33864408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61319790256128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67780518418432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚o„qé¿615 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 842975528448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 838908991594 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33864408 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18688582 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13462963 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 842975528448 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68662640656896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61319772253184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33839534 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18713454 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68662389278208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 227438592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15150954 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67780494478336 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60203882116986 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18713454 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66216139734016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65319107458560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19933499 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9659050 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63235927290384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚1280.721123005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4684885 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 939628376064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65319125461504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65319107458560 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 897941479936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72201553996288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64130504231264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65319125461504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897032275456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4684272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚799 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64129579224448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9659048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66217066941440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36072293 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16138794 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚73141425375744 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73141182372352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19908653 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66216139734016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14343935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16138792 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 897941479936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570¶†É÷“ÎO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚799 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64129579224448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9659048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚66217066941440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36072293 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16138794 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚73141425375744 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚73141182372352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19908653 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚66216139734016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14343935 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16138792 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 897941479936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66216139734016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65319107458560 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24HK—=Ã848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19933499 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9659050 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63235927290384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚1280.721123005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4684885 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1222 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 939628376064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65319125461504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65319107458560 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 897941479936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72201553996288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64130504231264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65319125461504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897032275456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4684272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640=KR+†2”- _wandbruntime‚1282  _runtime‚1282" = Ø&•ÒM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16158952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 940818359296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64209714092270 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚73232598060032 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19958356 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65400726544384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4690759 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788536487097621e+09 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16158954 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16158952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210661376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19933484 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66299807500800 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19933484 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64210639099294 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898153735680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36117310 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚799 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36092436 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72291803640832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234601472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16158954 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66298880280064 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚73232832661504 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63314927845568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899062953472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 899062953472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19958356 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14361248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63314944884928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 941029020672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36117310 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚799 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36092436 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72291803640832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234601472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16158954 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚66298880280064 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚73232832661504 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63314927845568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899062953472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 899062953472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19958356 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14361248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚624 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63314944884928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 941029020672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65400726544384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4690759 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788536487097621e+09 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16158954 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16158952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210661376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19933484 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185>?%²Ì2—- _wandbruntime‚1282  _runtime‚1282 bÌM<˜¶ 0loss/FRt4/assignment_loss‚3.5790665131626707 0loss/FBt2/assignment_loss‚0.3220398221097741 /loss/FRt3/detection_loss‚2.7527926709200883 trainer/global_step‚799 /loss/FRt3/assignment_loss‚4.593028894952826 2loss/SRqqt2/assignment_loss‚3.3798143598768444 epoch‚0 1loss/SRqqt3/assignment_loss‚2.335487731695175 1loss/SRqqt4/detection_loss‚1.1362050162539596 .loss/FRt1/detection_loss‚2.537108350678214 /loss/FRt1/assignment_loss‚5.934878062634241 0loss/FBt1/assignment_loss‚0.6436081528663635 loss/FBt3/assignment_loss‚0 1loss/SRqqt3/detection_loss‚1.9910736614465714 %loss/total_loss‚59.52694269184014 _runtime‚ 1282.41618328 _step‚63 .loss/FRt2/assignment_loss‚4.08961634238561 % _timestamp‚1.7885364871049037e+09 1loss/SRqqt1/assignment_loss‚3.284631684422493 0loss/SRqqt1/detection_loss‚2.781639431913694 -loss/FBt1/detection_loss‚4.26731915473938 /loss/FBt2/detection_loss‚0.5759146310830183 /loss/FBt4/detection_loss‚3.8493795543909073 .loss/FRt2/detection_loss‚2.546309714516004 0loss/SRqqt4/assignment_loss‚2.34910808631352 0loss/FBt4/assignment_loss‚0.7116799244831782 /loss/FRt4/detection_loss‚1.6940629870771688 0loss/SRqqt2/detection_loss‚3.456179294321272 .loss/FBt3/detection_loss‚0.715998649597168Ôwîù@™º 1loss/SRqqt1/assignment_loss‚3.284631684422493 0loss/SRqqt1/detection_loss‚2.781639431913694 -loss/FBt1/detection_loss‚4.26731915473938 /loss/FBt2/detection_loss‚0.5759146310830183 /loss/FBt4/detection_loss‚3.8493795543909073 .loss/FRt2/detection_loss‚2.546309714516004 0loss/SRqqt4/assignment_loss‚2.34910808631352 0loss/FBt4/assignment_loss‚0.7116799244831782 /loss/FRt4/detection_loss‚1.6940629870771688 0loss/SRqqt2/detection_loss‚3.456179294321272 .loss/FBt3/detection_loss‚0.715998649597168 0loss/FRt4/assignment_loss‚3.5790665131626707 0loss/FBt2/assignment_loss‚0.3220398221097741 /loss/FRt3/detection_loss‚2.7527926709200883 trainer/global_step‚799 /loss/FRt3/assignment_loss‚4.593028894952826 2loss/SRqqt2/assignment_loss‚3.3798143598768444 epoch‚0 1loss/SRqqt3/assignment_loss‚2.335487731695175 1loss/SRqqt4/detection_loss‚1.1362050162539596 .loss/FRt1/detection_loss‚2.537108350678214 /loss/FRt1/assignment_loss‚5.934878062634241 0loss/FBt1/assignment_loss‚0.6436081528663635 loss/FBt3/assignment_loss‚0 1loss/SRqqt3/detection_loss‚1.9910736614465714 %loss/total_loss‚59.52694269184014 _runtime‚ 1282.41618328 _step‚63 .loss/FRt2/assignment_loss‚4.08961634238561 % _timestamp‚1.7885364871049037e+09?˜ QO“š‚(:ˆ ®ÍëÔ§¶åÊ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èÿog›‚(:Ü ®ÍëÔÌðžÙ disk.dm-0.in‚ 3192.06640625 disk.dm-1.in‚0 disk.dm-1.out‚ 42.8671875# memory_percent‚95.1899528503418! proc.memory.rssMB‚ 2956.578125% disk./.usageGB‚1177.7859001159668 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 24703.2109375* proc.memory.percent‚18.045520782470703 cpu‚25.277497181468377 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 3192.06640625! disk.nvme0n1.in‚ 3192.06640625 disk.sda3.out‚ 42.8671875& proc.memory.availableMB‚ 788.078125 proc.cpu.threads‚17) disk./.usagePercent‚79.04219998497543 disk.sda2.in‚0 disk.drbd1260.out‚ 0.16796875$ disk.nvme0c0n1.out‚ 24703.2109375" disk.nvme0n1.out‚ 24703.2109375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 42.8671875 network.sent‚ 1.240189e+06 network.recv‚478431 disk.sda1.in‚0Pþs‘œ‚(:† ½ÍëÔä­ƒË gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.531 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¡´¡y‚(:î ½ÍëÔ ¢ÜÍ disk.drbd1260.in‚0 disk.drbd1260.out‚0.171875 network.recv‚483379! disk.nvme0n1.in‚ 3192.06640625 disk.sda3.in‚0 disk.sda3.out‚ 43.27734375 disk.sda1.in‚0 disk.dm-0.in‚ 3192.06640625 disk.sda2.out‚0$ memory_percent‚95.18845081329346) disk./.usagePercent‚79.04062374658712 disk.sda.out‚ 43.27734375 disk.dm-1.out‚ 43.27734375( disk.nvme0c0n1.out‚24934.46826171875( proc.memory.availableMB‚ 788.32421875# proc.memory.rssMB‚ 2956.58203125* proc.memory.percent‚18.045544624328613 cpu‚25.275796251782445% disk./.usageGB‚1177.7624130249023 disk.sda.in‚0# disk.nvme0c0n1.in‚ 3192.06640625 proc.cpu.threads‚17 network.sent‚ 1.289068e+06& disk.nvme0n1.out‚24934.46826171875 disk.sda1.out‚0# disk.dm-0.out‚24934.46826171875 disk.dm-1.in‚0 disk.sda2.in‚0†> ]’ž‚(:‡ ÌÍëÔª™îË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚166.851' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®4‚xŸ‚(:í ÌÍëÔ˜ÙÊÏ proc.cpu.threads‚17$ memory_percent‚95.18961906433105' proc.memory.availableMB‚ 788.1328125# proc.memory.rssMB‚ 2956.58203125 cpu‚25.273980717550064 network.recv‚487430# disk.dm-0.out‚25256.10498046875 disk.dm-1.in‚0 disk.dm-1.out‚ 43.62109375( disk.nvme0c0n1.out‚25256.10498046875& disk.nvme0n1.out‚25256.10498046875 disk.sda3.out‚ 43.62109375 disk.sda1.in‚0 disk.dm-0.in‚ 3343.75390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.171875# disk.nvme0c0n1.in‚ 3343.75390625! disk.nvme0n1.in‚ 3343.75390625 disk.sda1.out‚0 network.sent‚ 1.295003e+06) disk./.usagePercent‚79.04056051272616% disk./.usageGB‚1177.7614707946777 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 43.62109375* proc.memory.percent‚18.045544624328613ûsº‹“ ‚(:ˆ ÛÍëÔÂË gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž o¡‚(:ä ÛÍëÔŠ“Í disk.dm-1.out‚ 43.95703125$ memory_percent‚95.19014358520508# proc.memory.rssMB‚ 2956.66015625* proc.memory.percent‚18.046021461486816 disk.sda2.out‚0& proc.memory.availableMB‚ 788.046875 cpu‚25.27201042430566 proc.cpu.threads‚17 disk.sda3.in‚0 network.sent‚ 1.303249e+06 network.recv‚492126) disk./.usagePercent‚79.04214263928776% disk./.usageGB‚1177.7850456237793 disk.sda2.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 3343.75390625 disk.drbd1260.out‚ 0.17578125# disk.nvme0c0n1.in‚ 3343.75390625# disk.nvme0n1.out‚25574.09765625 disk.sda3.out‚ 43.95703125 disk.sda1.out‚0 disk.dm-0.in‚ 3343.75390625 disk.dm-0.out‚25574.09765625 disk.dm-1.in‚0% disk.nvme0c0n1.out‚25574.09765625 disk.sda.in‚0 disk.sda.out‚ 43.95703125 disk.sda1.in‚0sÛ<[2¢- _wandbruntime‚1348  _runtime‚1348ÅFÄ£¾ & lr-AdamW/pg2‚0.0011836090143056803 trainer/global_step‚849 $ _timestamp‚1.788536553002822e+09 _runtime‚1348.313850336 _step‚64 & lr-AdamW/pg1‚0.0011836090143056803ÝWȤ & lr-AdamW/pg2‚0.0011836090143056803 trainer/global_step‚849 $ _timestamp‚1.788536553002822e+09 _runtime‚1348.313850336 _step‚64 & lr-AdamW/pg1‚0.0011836090143056803@ˆVA2¥- _wandbruntime‚1348  _runtime‚1348Áð‹ó¦ËO ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 954028904960 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10262300 VHDevi—¯#×´&ceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249294848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17146792 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38300347 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70353181450240 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77711341575680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 998727711744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4980799 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69400079752704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 954028904960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 953119700480 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21153555 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69400079752704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67186807017584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67186789978224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15243099 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70354108657664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15242461 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 % _timestamp‚1.7885365530046263e+09 trainer/global_step‚849 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21153555 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70353181450240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69400061749760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 950454195988 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 998502356992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21178401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17146792 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21178401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17146794 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70354108657664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 _step‚65 _runtime‚ 1348.31588398 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68137261213572 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4980163 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 950454195988 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 998502356992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21178401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17146792 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21178401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17146794 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70354108657664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 _step‚65 _runtime‚ 1348.31588398 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68137261213572 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4980163 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70353181450240 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77711341575680 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 998727711744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4980799 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69400079752704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 954028904960AÚåz-2¨- _wandbruntime‚1349  _runtime‚1349ü«Ü&©ÖM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21178386 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76803088863232 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885365540554779e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70436849217024 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1225 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68217396081602 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21203258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚66 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15260273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10274363 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17166952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67265824612128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208564224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68216471074578 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17166954 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38370212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17166954 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 954241160704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76803064923136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚1349.367053453 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38345338 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69481698838528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38345338 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67265807572768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69481680835584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69481698838528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 951571469474 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 954241160704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38370212 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17166952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232504320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69481680835584 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17166954 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38370212 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17166954 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 954241160704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76803064923136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚1349.367053453 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38345338 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69481698838528 G:DeviceStatsMonitor.on_train_batch_end/allocation.Š×Ø;šall.freed‚38345338 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67265807572768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69481680835584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69481698838528 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 951571469474 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 954241160704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38370212 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17166952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232504320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69481680835584 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21178386 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76803088863232 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885365540554779e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70436849217024 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1225 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68217396081602 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21203258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚66 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15260273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10274363 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17166952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67265824612128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208564224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68216471074578 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1005423483640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71137686744784 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71137669705424 _runtime‚1416.043648878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5276637 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1057737539072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73481016040960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 _step‚69 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚899 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16142827 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16142185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚73481016040960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18154794 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10865550 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74491150373888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74490223166464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1009207125504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40553249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18154794 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1010116329984 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82281861980160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81224148381184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22423303 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72144018195880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1009207125504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1010116329984 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72143093189064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1006331451096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18154792ùFï’3½ØO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74491150373888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1005423483640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71137686744784 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71137669705424 _runtime‚1416.043648878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5276637 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212771840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1057737539072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚73481016040960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 _step‚69 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚899 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16142827 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16142185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚73481016040960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18154794 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10865550 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74491150373888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74490223166464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1009207125504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40553249 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 MGDeviceStatsMonitor.on_train_bat˜Á¸œ«ch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18154794 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1010116329984 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82281861980160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81224148381184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22423303 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72144018195880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1009207125504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1010116329984 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72143093189064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1006331451096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18154792 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74490223166464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10865548 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5277277 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18154792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1057950310912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1010328585728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16160128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1007448724582 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18174952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10877615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22423288 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1011237803520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40623114 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10877613 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82373268908544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74573890933248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5283149 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82373513995776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245087232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22423288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71216704339328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71216687299968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1011237803520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81314374085632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1058918763008 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72224153063910 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73562635126784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73562635126784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788536622052366e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73562653129728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73562653129728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22448160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1059139910144 _step‚70 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16160764 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72223228056886 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40598240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 trainer/global_step‚899 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18174954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74573890933248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚1417.364834237 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22448160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1006540756918 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1010328585728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5282515 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18174954 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18174952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144“ ZÕç&ÀáM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1225 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40623114 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81314350145536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1007448724582 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18174952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10877615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22423288 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1011237803520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40623114 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10877613 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82373268908544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74573890933248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5283149 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82373513995776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245087232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22423288 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71216704339328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71216687299968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1011237803520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81314374085632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1058918763008 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72224153063910 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73562635126784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚73562635126784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788536622052366e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚73562653129728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73562653129728 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22448160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1059139910144 _step‚70 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16160764 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72223228056886 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40598240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 trainer/global_step‚899 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18174954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74573890933248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚1417.364834237 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22448160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1006540756918 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1010328585728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5282515 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18174954 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18174952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1262 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1010328585728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16160128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23668205 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76150775178188 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19162792 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42830999 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1062208706204 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1485.558458495 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78627264882688 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77561988335104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17041725 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76149850171372 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75088549432624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1066203755008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1065294550528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11468798 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11468800 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86852087194112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5573588 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 trainer/global_step‚949 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23643359š8ÊÞ'ÓØO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1065294550528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚663 _step‚73 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19162792 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75088566471984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78628192090112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85735409663488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19162794 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 243003392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1116896593920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23668205 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚661 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5572927 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11468800 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86852087194112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5573588 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1262 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 trainer/global_step‚949 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23643359 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1066203755008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23668205 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76150775178188 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19162792 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42830999 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1062208706204 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1485.558458495 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1225 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78627264882688 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77561988335104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17041725 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76149850171372 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75088549432624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1066203755008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1065294550528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11468798 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17059557 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚74 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1066416010752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19182952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85825659308032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885366913988655e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1063325979690 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11480863 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1067325228544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17060208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78710932649472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5579343 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23668190 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77643607420928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77643589417984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85825635367936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1118080227840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚949 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1067325228544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42876016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221147136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76230910046218 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23668190 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42876016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11480865 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42851142 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17059557 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚74 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1066416010752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19182952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85825659308032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885366913988655e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚649 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1063325979690 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11480863 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1067325228544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17060208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78710932649472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5579343 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23668190 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚651 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77643607420928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77643589417984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85825635367936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1118080227840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚949 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1067325228544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42876016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221147136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76230910046218 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23668190 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42876016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11480865 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42851142 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82765233806336 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90246718825984 trainer/global_step‚999 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82764306598912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80156607153680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81642924623360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1226 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0`˜èßàçØO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45083901 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45059053 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1121381975552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5868563 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1175900916224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1122291180032 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24888261 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20170794 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17941266 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82765233806336 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90246718825984 trainer/global_step‚999 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82764306598912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80156607153680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81642924623360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1226 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1122291180032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24913107 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1263 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5869216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20170794 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82765233806336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91422383030272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79039446199184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82764306598912 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17940611 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24888261 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1175688144384 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80157532160496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24913107 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81642942626304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90246694885888MÐËÀ2è- _wandbruntime‚1558  _runtime‚1558ŠYeã&éÝM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81724561712128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24937964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82847974365696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1226 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81724543709184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45128918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5875002 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91514034126336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45104044 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885367636278956e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20190952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚999 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1123412653568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17958473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80237667028526 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79118463793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80236742021502 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _step‚78 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24913092 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82847974365696 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81724543709184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20190952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91513793233408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81724561712128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1122503435776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1118295267134 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 1558.93972036 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5874360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45104044 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ðÔç&êáM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17958473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80237667028526 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79118463793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80236742021502 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _step‚78 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24913092 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82847974365696 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81724543709184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20190952 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚91513793233408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚81724561712128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1122503435776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1118295267134 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 1558.93972036 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5874360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45104044 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240892928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81724561712128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24937964 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82847974365696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1226 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81724543709184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45128918 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5875002 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91514034126336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45104044 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885367636278956e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20190952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚999 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1123412653568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚644 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152N¤vRŠ2ë- _wandbruntime‚1558  _runtime‚1558>üïúSìÍ /loss/FRt3/assignment_loss‚5.916798634188516 % _timestamp‚1.7885367636304038e+09 1loss/SRqqt3/assignment_loss‚2.461759875560629 0loss/SRqqt1/detection_loss‚2.662354907164207 &loss/total_loss‚56.847126036631295 /loss/FRt2/detection_loss‚2.3157155896936144 /loss/FRt4/detection_loss‚2.0151669258123297 0loss/SRqqt2/detection_loss‚2.727146443198709 /loss/FBt2/detection_loss‚0.8321836196020693 _runtime‚1558.941218735 0loss/FBt3/assignment_loss‚0.6486795246601105 0loss/FBt4/assignment_loss‚0.4163851924240589 1loss/SRqqt1/assignment_loss‚2.363086993877704 0loss/FBt2/assignment_loss‚0.2658240898471617 /loss/FRt3/detection_loss‚3.1260863159384047 1loss/SRqqt4/detection_loss‚1.0297497070648454 epoch‚0 trainer/global_step‚999 0loss/FRt1/assignment_loss‚4.8502136556113635 0loss/SRqqt4/assignment_loss‚2.06783259530698 /loss/FBt4/detection_loss‚2.6369048257668815 0loss/SRqqt3/detection_loss‚2.686960010692991 /loss/FBt1/detection_loss‚2.5727431819989133 _step‚79 /loss/FBt3/detection_loss‚4.2028100192546844 .loss/FRt1/detection_loss‚2.614956054745651 0loss/FRt2/assignment_loss‚2.8852787835257394 /loss/FRt4/assignment_loss‚2.982492033392191 1loss/SRqqt2/assignment_loss‚1.945342901874991 0loss/FBt1/assignment_loss‚0.6206541554285493ÒžëWíÑ _step‚79 /loss/FBt3/detection_loss‚4.2028100192546844 .loss/FRt1/detection_loss‚2.614956054745651 0loss/FRt2/assignment_loss‚2.8852787835257394 /loss/FRt4/assignment_loss‚2.982492033392191 1loss/SRqqt2/assignment_loss‚1.945342901874991 0loss/FBt1/assignment_loss‚0.6206541554285493 /loss/FRt3/assignment_loss‚5.916798634188516 % _timestamp‚1.7885367636304038e+09 1loss/SRqqt3/assignment_loss‚2.461759875560629 0loss/SRqqt1/detection_loss‚2.662354907164207 &loss/total_loss‚56.847126036631295 /loss/FRt2/detection_loss‚2.3157155896936144 /loss/FRt4/detection_loss‚2.0151669258123297 0loss/SRqqt2/detection_loss‚2.727146443198709 /loss/FBt2/detection_loss‚0.8321836196020693 _runtime‚1558.941218735 0loss/FBt3/assignment_loss‚0.6486795246601105 0loss/FBt4/assignment_loss‚0.4163851924240589 1loss/SRqqt1/assignment_loss‚2.363086993877704 0loss/FBt2/assignment_loss‚0.2658240898471617 /loss/FRt3/detection_loss‚3.1260863159384047 1loss/SRqqt4/detection_loss‚1.0297497070648454 epoch‚0 trainer/global_step‚999 0loss/FRt1/assignment_loss‚4.8502136556113635 0loss/SRqqt4/assignment_loss‚2.06783259530698 /loss/FBt4/detection_loss‚2.6369048257668815 0loss/SRqqt3/detection_loss‚2.686960010692991 /loss/FBt1/detection_loss‚2.5727431819989133Oó±h7“î‚(:ˆ ¼ÏëÔŠí´Ë gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.602' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÛE„kï‚(:à ¼ÏëÔ­½Ð network.sent‚ 1.527769e+06 network.recv‚555845 disk.dm-0.out‚29922.49609375 disk.dm-1.in‚0 disk.drbd1260.in‚0# disk.nvme0n1.out‚29922.49609375 disk.sda1.in‚0( proc.memory.availableMB‚ 787.23828125 disk.dm-0.in‚ 3741.078125 disk.drbd1260.out‚ 0.20703125! disk.nvme0c0n1.in‚ 3741.078125% disk.nvme0c0n1.out‚29922.49609375 disk.nvme0n1.in‚ 3741.078125 disk.sda3.in‚0$ memory_percent‚95.19507884979248# proc.memory.rssMB‚ 2956.66796875 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 51.25390625 disk.sda1.out‚0* proc.memory.percent‚18.046069145202637) disk./.usagePercent‚79.04252972267953$ disk./.usageGB‚1177.790813446045 disk.dm-1.out‚ 51.25390625 disk.sda2.in‚0 disk.sda3.out‚ 51.25390625 cpu‚25.251409248389347 proc.cpu.threads‚17–îëž’ð‚(:‡ ËÏëÔµìñË gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚170.315' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.063 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Ó°K^ñ‚(:Ó ËÏëÔÜïüÎ disk.dm-1.in‚0 disk.sda3.out‚51.84375 network.recv‚561608$ disk./.usageGB‚1177.767406463623 disk.sda2.in‚0& disk.nvme0c0n1.out‚30254.083984375 disk.nvme0n1.in‚ 3882.15625$ disk.nvme0n1.out‚30254.083984375 disk.sda.out‚51.84375 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 3882.15625 disk.dm-1.out‚51.84375& proc.memory.availableMB‚ 787.703125# proc.memory.rssMB‚ 2956.66796875 network.sent‚ 1.564738e+06 disk.drbd1260.in‚0$ memory_percent‚95.19224166870117* proc.memory.percent‚18.046069145202637 cpu‚25.24990595286105 proc.cpu.threads‚17) disk./.usagePercent‚79.04095886044942 disk.drbd1260.out‚ 0.2109375 disk.nvme0c0n1.in‚ 3882.15625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0! disk.dm-0.out‚30254.083984375£"¬žò‚(:“ ÚÏëÔâõóÊ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.918' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.183600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïG3ÔWó‚(:Ì ÚÏëÔò߶Í disk.sda1.out‚0 disk.dm-0.in‚ 3882.15625! disk.dm-0.out‚30513.197265625 disk.dm-1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 3882.15625$ disk.nvme0n1.out‚30513.197265625 disk.sda3.in‚0$ memory_percent‚95.19224166870117& proc.memory.availableMB‚ 787.703125* proc.memory.percent‚18.046069145202637 cpu‚25.248886998755378 disk.sda3.out‚52.25 network.sent‚ 1.569691e+06 network.recv‚564960) disk./.usagePercent‚79.04253509883775 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2109375 disk.nvme0c0n1.in‚ 3882.15625 disk.dm-1.out‚52.25& disk.nvme0c0n1.out‚30513.197265625 disk.sda1.in‚0# proc.memory.rssMB‚ 2956.66796875 proc.cpu.threads‚17% disk./.usageGB‚1177.7908935546875 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚52.25nÚ>nžô‚(:“ éÏëÔìüôË gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.508' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.701600000000003 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~s®^@õ‚(:Ü éÏëÔœž¦Ð$ disk.nvme0n1.out‚31037.7363281259¹6 ' disk.sda1.in‚0' proc.memory.availableMB‚ 787.4453125* proc.memory.percent‚18.046069145202637 cpu‚25.247748733157536$ disk./.usageGB‚1177.767406463623& disk.nvme0c0n1.out‚31037.736328125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 52.7421875 disk.sda1.out‚0 disk.dm-0.in‚ 3882.15625 disk.dm-1.in‚0) disk./.usagePercent‚79.04095886044942 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.21484375 disk.nvme0n1.in‚ 3882.15625 disk.sda3.out‚ 52.7421875! disk.dm-0.out‚31037.736328125 disk.dm-1.out‚ 52.7421875$ memory_percent‚95.19381523132324 network.sent‚ 1.576718e+06 network.recv‚568480 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 3882.15625# proc.memory.rssMB‚ 2956.66796875 proc.cpu.threads‚17 disk.sda2.out‚0¹ïŽìžö‚(:“ øÏëÔ°¾³Ë gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.211' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.842200000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÊUŽf÷‚(:Û øÏëÔŒ®ÊÌ disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 3882.15625& disk.nvme0c0n1.out‚31359.435546875 disk.sda.out‚ 53.33984375$ proc.memory.availableMB‚787.1875# proc.memory.rssMB‚ 2956.66796875* proc.memory.percent‚18.046069145202637 network.sent‚ 1.58271e+06 disk.dm-0.in‚ 3882.15625 disk.dm-1.out‚ 53.33984375 disk.sda2.in‚0 disk.nvme0n1.in‚ 3882.15625 disk.sda3.in‚0 disk.sda1.in‚0 cpu‚25.24668634806037 network.recv‚572833 disk.drbd1260.in‚0$ disk.nvme0n1.out‚31359.435546875 disk.sda3.out‚ 53.33984375) disk./.usagePercent‚78.99053382445273! disk.dm-0.out‚31359.435546875 disk.sda2.out‚0 disk.drbd1260.out‚ 0.21484375 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚95.19538879394531 proc.cpu.threads‚17% disk./.usageGB‚1177.0160369873047ë¯lŸ2ø- _wandbruntime‚1624  _runtime‚1624“"ÙÄù¾ % lr-AdamW/pg2‚0.001462433281515499 trainer/global_step‚1049 % _timestamp‚1.7885368293544934e+09 _runtime‚1624.665507115 _step‚80 % lr-AdamW/pg1‚0.001462433281515499m¾IèÈú _runtime‚1624.665507115 _step‚80 % lr-AdamW/pg1‚0.001462433281515499 % lr-AdamW/pg2‚0.001462433281515499 trainer/global_step‚1049 % _timestamp‚1.7885368293544934e+09PŸØõÕ2û- _wandbruntime‚1624  _runtime‚1624ƒ`ÖçÛ'üÕO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26133163 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚644 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26158009 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21178792 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95992874775552 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18838843 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94757980108288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82990325926384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚1049 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚85723878914560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47336803 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18839487 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6164187 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚642 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26158009 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21178794 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85723878914560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6163545 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86901348315136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1178378605056 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1173963216420 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85723896917504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206480384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85723896917504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94758004048384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86902275522560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1624.667527314 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177469400576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95993105196032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230420480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1234894667264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21178794 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85723878914560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6163545 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86901348315136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1178378605056 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1173963216420 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚85723896917504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206480384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85723896917504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94758004048384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86902275522560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1624.667527314 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177469400576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95993105196032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230420480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1234894667264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95992874775552 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18838843 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94757980108288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82990325926384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615Q£Vž.2þ- _wandbruntime‚1626  _runtime‚1626 Øâ¬%ÿßM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚85805498000384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21198952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94848205812736 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83069326481568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1178590860800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85805516003328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86985016081920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 227438592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1236067126784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84243499003810 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47356946 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47356946 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94848229752832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚1626.513557694 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86984088861184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96084272939520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21198954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179500078592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21198954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6169455 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1236294565376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18857484 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6170119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18856818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21198952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84244424010834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1175080489906 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1174172522242 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserveãjMB9d_bytes.large_pool.peak‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47381820 _step‚82 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85805516003328 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96084524318208c¯òé&€ãM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26182866 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26182866 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26157994 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1236294565376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18857484 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6170119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18856818 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21198952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84244424010834 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1175080489906 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1174172522242 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47381820 _step‚82 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85805516003328 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96084524318208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚85805498000384 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21198952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94848205812736 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83069326481568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1178590860800 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85805516003328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86985016081920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 227438592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1236067126784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84243499003810 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47356946 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47356946 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94848229752832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚1626.513557694 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86984088861184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96084272939520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21198954 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179500078592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21198954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6169455 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91038390031360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91038390031360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86941205653584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89804851208704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99269289270784 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1294330700800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1229840471528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88170121118296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22186794 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88171046125112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1233556825600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13278548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚85 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27402911 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1233556825600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6461901 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1228932504072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _runtime‚1692.105223374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6461243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Ø $±|‘ÛO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88170121118296 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22186794 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88171046125112 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1233556825600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13278548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚85 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27402911 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1233556825600 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6461901 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1228932504072 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _runtime‚1692.105223374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6461243 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22186794 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91039317238784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1294115831808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91038390031360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91038390031360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86941205653584 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89804851208704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99269289270784 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1294330700800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1229840471528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0U š2’- _wandbruntime‚1693  _runtime‚1693/½iDç&“áM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49634722 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1234678285824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13290613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99359514975232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1235587503616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27402896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99359491035136 trainer/global_step‚1099 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88250255986118 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91122057798144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89886470294528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22206952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19758440 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89886452291584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27402896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885368979796436e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49609848 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚1693.291393975 _step‚86 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91121130577408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1234678285824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19757790 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100654790586368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91122057798144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88251180993142 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13290615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1235587503616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1295518601216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87020223248128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89886470294528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6467825 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100655033576448BÝóë&”åM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87020223248128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89886470294528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6467825 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100655033576448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49634722 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1234678285824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13290613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99359514975232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 219049984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1235587503616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242990080 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27402896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99359491035136 trainer/global_step‚1099 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88250255986118 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91122057798144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89886470294528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22206952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19758440 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89886452291584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27402896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885368979796436e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49609848 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚1693.291393975 _step‚86 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91121130577408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1234678285824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19757790 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100654790586368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91122057798144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88251180993142 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13290615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1235587503616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1295518601216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872Vïʨ€2•- _wandbruntime‚1693  _runtime‚1693j·‚X–Ò /loss/FRt1/assignment_loss‚5.430616985667836 2loss/SRqqt2/assignment_loss‚1.6326641368077082 2loss/SRqqt3/assignment_loss‚2.0987917963535554 1loss/FBt1/assignment_loss‚0.36668586180930746 /loss/FRt1/detection_loss‚3.1140502273584856 0loss/SRqqt2/detection_loss‚2.868154218968223 .loss/FBt2/detection_loss‚1.051218938853716 .loss/FBt4/detection_loss‚2.304001443915897 % _timestamp‚1.7885368979825792e+09 _step‚87 0loss/FBt4/assignment_loss‚0.4910828740749922 .loss/FRt3/detection_loss‚3.325804196871244 1loss/SRqqt4/detection_loss‚0.9662945303767919 &loss/total_loss‚55.537606177299466 trainer/global_step‚1099 0loss/FRt2/assignment_loss‚3.1949213246504464 0loss/FRt4/assignment_loss‚2.6326524251037173 0loss/FBt3/assignment_loss‚0.4300411586445982 /loss/FBt3/detection_loss‚3.1180847942829133 1loss/SRqqt1/assignment_loss‚3.087366004784902 2loss/SRqqt4/assignment_loss‚1.5870465689897537 1loss/SRqqt3/detection_loss‚2.6168164264771248 /loss/FRt2/detection_loss‚2.1212843228821403 .loss/FRt4/detection_loss‚1.695875892881304 epoch‚0 0loss/FRt3/assignment_loss‚5.2255658736595745 1loss/SRqqt1/detection_loss‚2.6890709524353347 _runtime‚1693.293411448 0loss/FBt2/assignment_loss‚0.2306479786928884 .loss/FBt1/detection_loss‚3.258867242757012Éu)\—Ö 0loss/FRt3/assignment_loss‚5.2255658736595745 1loss/SRqqt1/detection_loss‚2.6890709524353347 _runtime‚1693.293411448 0loss/FBt2/assignment_loss‚0.2306479786928884 .loss/FBt1/detection_loss‚3.258867242757012 /loss/FRt1/assignment_loss‚5.430616985667836 2loss/SRqqt2/assignment_loss‚1.6326641368077082 2loss/SRqqt3/assignment_loss‚2.0987917963535554 1loss/FBt1/assignment_loss‚0.36668586180930746 /loss/FRt1/detection_loss‚3.1140502273584856 0loss/SRqqt2/detection_loss‚2.868154218968223 .loss/FBt2/detection_loss‚1.051218938853716 .loss/FBt4/detection_loss‚2.304001443915897 % _timestamp‚1.7885368979825792e+09 _step‚87 0loss/FBt4/assignment_loss‚0.4910828740749922 .loss/FRt3/detection_loss‚3.325804196871244 1loss/SRqqt4/detection_loss‚0.9662945303767919 &loss/total_loss‚55.537606177299466 trainer/global_step‚1099 0loss/FRt2/assignment_loss‚3.1949213246504464 0loss/FRt4/assignment_loss‚2.6326524251037173 0loss/FBt3/assignment_loss‚0.4300411586445982 /loss/FBt3/detection_loss‚3.1180847942829133 1loss/SRqqt1/assignment_loss‚3.087366004784902 2loss/SRqqt4/assignment_loss‚1.5870465689897537 1loss/SRqqt3/detection_loss‚2.6168164264771248 /loss/FRt2/detection_loss‚2.1212843228821403 .loss/FRt4/detection_loss‚1.695875892881304 epoch‚0WR‡‡|“˜‚(:ˆ ÃÐëÔâìÍ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.308' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k>'j™‚(:ß ÃÐëÔ°Ö´Ñ% disk./.usageGB‚1177.1186027526855 disk.drbd1260.out‚ 0.2265625# proc.memory.rssMB‚ 2956.66796875 cpu‚25.241234693515995 disk.dm-0.in‚ 4178.3359375 disk.dm-0.out‚32540.66796875 disk.sda2.out‚0# disk.nvme0n1.out‚32540.66796875 disk.sda.out‚ 56.0390625( proc.memory.availableMB‚ 787.66015625* proc.memory.percent‚18.046069145202637 network.sent‚ 1.65338e+06) disk./.usagePercent‚78.99741709902585 disk.dm-1.in‚0 disk.dm-1.out‚ 56.0390625 disk.sda2.in‚0 disk.drbd1260.in‚0% disk.nvme0c0n1.out‚32540.66796875 disk.sda3.out‚ 56.0390625" disk.nvme0c0n1.in‚ 4178.3359375 disk.nvme0n1.in‚ 4178.3359375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.19250392913818 proc.cpu.threads‚17 network.recv‚594903É°(h“š‚(:ˆ ÒÐëÔ¼ãÌ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.969' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÎxPk›‚(:à ÒÐëÔ–ô£Ð disk.nvme0n1.in‚ 4178.3359375 disk.sda3.out‚ 56.5859375* proc.memory.percent‚18.046069145202637 cpu‚25.240490730524623 network.recv‚599609" disk.nvme0c0n1.in‚ 4178.3359375# disk.nvme0n1.out‚32948.30859375 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 4178.3359375 disk.dm-1.out‚ 56.5859375) disk./.usagePercent‚78.99761473684228 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 56.5859375 disk.dm-1.in‚0# proc.memory.rssMB‚ 2956.66796875 proc.cpu.threads‚17 network.sent‚ 1.702345e+06% disk.nvme0c0n1.out‚32948.30859375 disk.sda.in‚0 disk.dm-0.out‚32948.30859375$ memory_percent‚95.19250392913818( proc.memory.availableMB‚ 787.66015625% disk./.usageGB‚1177.1215476989746 disk.drbd1260.out‚ 0.2265625äÊÎ’œ‚(:‡ áÐëÔßÐ’Ë gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.155' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.031 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*ïÿÙÉ‚(:â áÐëÔþÔÍÏ disk.dm-1.out‚ 57.07421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.23046875" disk.nvme0c0n1.in‚ 4178.3359375$ disk.nvme0n1.out‚33145.697265625 disk.sda1.in‚0# proc.memory.rssMB‚ 2956.66796875! disk.dm-0.out‚33145.697265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 cpu‚25.239321103766805 network.recv‚603398& disk.nvme0c0n1.out‚33145.697265625 disk.nvme0n1.in‚ ÒÓUƤ4178.3359375 disk.sda3.in‚0# memory_percent‚95.1925277709961* proc.memory.percent‚18.046069145202637 proc.cpu.threads‚17 network.sent‚ 1.707362e+06$ disk./.usageGB‚1177.013584136963 disk.dm-0.in‚ 4178.3359375 disk.dm-1.in‚0 disk.sda3.out‚ 57.07421875 disk.sda.in‚0 disk.sda.out‚ 57.07421875% proc.memory.availableMB‚ 787.65625) disk./.usagePercent‚78.99036921160821ª±Aˆ’ž‚(:‡ ðÐëÔ室Ì gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’zü@mŸ‚(:â ðÐëÔ ÃÔÎ network.sent‚ 1.713271e+06 network.recv‚608074" disk.nvme0c0n1.in‚ 4178.3359375 disk.sda1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1177.0137634277344& disk.nvme0c0n1.out‚33236.650390625 disk.nvme0n1.in‚ 4178.3359375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 4178.3359375! disk.dm-0.out‚33236.650390625 disk.dm-1.in‚0) disk./.usagePercent‚78.99038124396232 disk.drbd1260.out‚ 0.23046875$ disk.nvme0n1.out‚33236.650390625 disk.sda3.in‚0# proc.memory.availableMB‚787.125# proc.memory.rssMB‚ 2956.66796875* proc.memory.percent‚18.046069145202637 cpu‚25.238179848150082 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 57.56640625 disk.sda.out‚ 57.56640625 disk.dm-1.out‚ 57.56640625$ memory_percent‚95.19577026367188rÔ:‘ ‚(:† ÿÐëÔúïÔÍ gpu.0.gpu‚10.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚166.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V€m¡‚(:â ÿÐëÔ«ÖêÑ network.sent‚ 1.721355e+06 disk.sda.out‚ 57.91796875 disk.sda1.in‚0! disk.nvme0n1.in‚ 4208.46484375$ memory_percent‚95.19121646881104( proc.memory.availableMB‚ 787.87109375* proc.memory.percent‚18.046069145202637 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 57.91796875 disk.sda2.out‚0 disk.drbd1260.out‚0.234375 proc.cpu.threads‚17 network.recv‚612453 disk.sda3.in‚0 disk.dm-0.in‚ 4208.46484375 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 4208.46484375 cpu‚25.237215420848194) disk./.usagePercent‚78.99038738814313% disk./.usageGB‚1177.0138549804688 disk.sda3.out‚ 57.91796875 disk.dm-0.out‚ 33548.3984375$ disk.nvme0c0n1.out‚ 33548.3984375" disk.nvme0n1.out‚ 33548.3984375# proc.memory.rssMB‚ 2956.66796875ÄlçÒ2¢- _wandbruntime‚1759  _runtime‚1759YF±ïÆ£À & lr-AdamW/pg2‚0.0016018454151204083 trainer/global_step‚1149 % _timestamp‚1.7885369642244058e+09 _runtime‚1759.535375654 _step‚88 & lr-AdamW/pg1‚0.0016018454151204083w¢ëÙÊ¤Ä & lr-AdamW/pg2‚0.0016018454151204083 trainer/global_step‚1149 % _timestamp‚1.7885369642244058e+09 _runtime‚1759.535375654 _step‚88 & lr-AdamW/pg1‚0.0016018454151204083X?­È2¥- _wandbruntime‚1759  _runtime‚1759¦7¯qß'¦ÙO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28647813 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95176358955008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6758759 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90892085380784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1289644250624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105134005590016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95175431747584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90892068341424 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28647813 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103780574493184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93885805499904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105133768878080 _step‚89 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1284809759180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23194794 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20640559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23194792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1290553455104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20639899 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95175431747584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1289644250624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93885787496960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103780550553088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93885805499904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚1149 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23194794 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23194792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚93885787496960 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28647813 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95176358955008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6758759 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 254704640 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90892085380784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1289644250624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105134005590016 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚95175431747584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90892068341424 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927207424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28647813 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103780574493184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚93885805499904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105133768878080 _step‚89 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1284809759180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23194794 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20640559 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚660 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23194792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236711936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1290553455104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20639899 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95175431747584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1289644250624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909204480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93885787496960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103780550553088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14134971392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93885805499904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚1149 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152YÍÝ’g2¨- _wandbruntime‚1760  _runtime‚1760Ø­p"©ãM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93967424585728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1290765710848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51862750 _step‚90 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1291674928640 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51887624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28672670 % _timestamp‚1.7885369653514082e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28647798 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1286835000122 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1285927032458 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20658543 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28672670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95259099514368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105225425442304 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23214954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6764678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103870800197632 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28647798 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103870776257536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95259099514368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90971102975328 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20657888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1760.662841901 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51887624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1291674928640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ICDeviceStatsMonitor.on_train_batch_end/allocation.largeSГÍ_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1354625244672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51887624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28672670 % _timestamp‚1.7885369653514082e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28647798 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1286835000122 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1285927032458 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20658543 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28672670 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95259099514368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105225425442304 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23214954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6764678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103870800197632 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28647798 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103870776257536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95259099514368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90971102975328 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20657888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1760.662841901 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51887624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1291674928640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909217792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1354625244672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99312473463808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1412544866304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94842965107984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1345731675648 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99312473463808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97966741788160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚109704159481344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109704404581888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7055321 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24202794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29867869 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97966759791104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1341594981744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 % _timestamp‚1.7885370316321278e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54095509 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97966759791104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108291835775488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚1199 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99313400671232 _runtime‚1826.943712937 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29892715 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14485048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24202794 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29892715 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7054645 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14485050 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1340687014288 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1346640880128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1346640880128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1412323705856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24202792õäÜOð »ÝO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925006816 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1346640880128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1346640880128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1412323705856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24202792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927207424 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684408973 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99313400671232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99312473463808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1412544866304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94842965107984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1345731675648 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99312473463808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97966741788160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚109704159481344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907967456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 90ã¢Çó9204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚109704404581888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7055321 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189001216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24202794 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29867869 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97966759791104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1341594981744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187720845 % _timestamp‚1.7885370316321278e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54095509 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97966759791104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189001216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245100544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108291835775488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚1199 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99313400671232 _runtime‚1826.943712937 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14134971392 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29892715 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14485048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24202794 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29892715 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7054645 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14485050 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1340687014288]}w2¼- _wandbruntime‚1828  _runtime‚1828ªcæÊé&½ãM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228310016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99396141230592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21558365 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885370332407334e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108382061479936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29917572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96263769950734 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54115652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98048378876928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54115652 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54140526 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96264694957758 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1342712255230 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚1199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7061250 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109795818602496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29892700 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1346853135872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109795590292480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1346853135872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108382085420032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94921965663168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1347762353664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14497113 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94921982702528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21557702 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24222952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14497115 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204369920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚1828.554236669 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24222954 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1413733182464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99396141230592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7060589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98048360873984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98048378876928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1341804287566 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1347762353664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 0­í&¾çM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684408973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927220736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1347762353664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228310016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99396141230592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21558365 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885370332407334e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907967664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108382061479936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29917572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925007024 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96263769950734 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54115652 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98048378876928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54115652 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 254704640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54140526 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96264694957758 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1342712255230 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 trainer/global_step‚1199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7061250 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚109795818602496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29892700 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909217792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1346853135872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚109795590292480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187720845 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1346853135872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108382085420032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94921965663168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1347762353664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14497113 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94921982702528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21557702 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24222952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14497115 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204369920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚1828.554236669 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24222954 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1413733182464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99396141230592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7060589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14134971392 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927220736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189001216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98048360873984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14134971392 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98048378876928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1341804287566 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185^ ×2¿- _wandbruntime‚1828  _runtime‚1828s_o\ÀÖ /loss/FRt2/assignment_loss‚4.033747577353528 1loss/SRqqt2/detection_loss‚3.2541379292805988 /loss/FBt3/detection_loss‚2.7950154463450114 .loss/FRt1/detection_loss‚2.786126446723938 1loss/SRqqt1/detection_loss‚2.6540273825327554 2loss/SRqqt2/assignment_loss‚1.7589130078752835 2loss/SRqqt3/assignment_loss‚1.7900313364389615 0loss/FBt1/assignment_loss‚0.2778094242479416 2loss/SRqqt1/assignment_loss‚2.6333929929468365 1loss/FBt3/assignment_loss‚0.25142060202973276 1loss/SRqqt3/detection_loss‚2.3862964724883056 &loss/total_loss‚53.800850043705694 epoch‚0 _runtime‚1828.557384943 2loss/SRqqt4/assignment_loss‚1.3391437981691625 0loss/FBt2/assignment_loss‚0.1714391656505124 /loss/FBt4/detection_loss‚3.6031946897506715 trainer/global_step‚1199 0loss/FRt1/assignment_loss‚4.7378564425877165 0loss/FRt4/assignment_loss‚2.4532734030677426 /loss/FBt2/detection_loss‚0.8824226828292012 _step‚95 /loss/FRt3/assignment_loss‚4.641289631525676 /loss/FRt3/detection_loss‚3.3699822965122403 1loss/SRqqt4/detection_loss‚1.0826149974495938 /loss/FBt1/detection_loss‚2.0925805138217077 /loss/FBt4/assignment_loss‚0.542330215126276 /loss/FRt2/detection_loss‚2.4098429454392507 /loss/FRt4/detection_loss‚1.8539606435130531 % _timestamp‚1.7885370332463448e+09Ò„ÃÐ`ÁÚ 0loss/FBt1/assignment_loss‚0.2778094242479416 2loss/SRqqt1/assignment_loss‚2.6333929929468365 1loss/FBt3/assignment_loss‚0.25142060202973276 1loss/SRqqt3/detection_loss‚2.3862964724883056 &loss/total_loss‚53.800850043705694 epoch‚0 _runtime‚1828.557384943 2loss/SRqqt4/assignment_loss‚1.3391437981691625 0loss/FBt2/assignment_loss‚0.1714391656505124 /loss/FBt4/detection_loss‚3.6031946897506715 trainer/global_step‚1199 0loss/FRt1/assignment_loss‚4.7378564425877165 0loss/FRt4/assignment_loss‚2.4532734030677426 /loss/FBt2/detection_loss‚0.8824226828292012 _step‚95 /loss/FRt3/assignment_loss‚4.641289631525676 /loss/FRt3/detection_loss‚3.3699822965122403 1loss/SRqqt4/detection_loss‚1.0826149974495938 /loss/FBt1/detection_loss‚2.0925805138217077 /loss/FBt4/assignment_loss‚0.542330215126276 /loss/FRt2/detection_loss‚2.4098429454392507 /loss/FRt4/detection_loss‚1.8539606435130531 % _timestamp‚1.7885370332463448e+09 /loss/FRt2/assignment_loss‚4.033747577353528 1loss/SRqqt2/detection_loss‚3.2541379292805988 /loss/FBt3/detection_loss‚2.7950154463450114 .loss/FRt1/detection_loss‚2.786126446723938 1loss/SRqqt1/detection_loss‚2.6540273825327554 2loss/SRqqt2/assignment_loss‚1.7589130078752835 2loss/SRqqt3/assignment_loss‚1.7900313364389615_½>/Š‘‚(:† ÊÑëÔ”‚ŒË gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚126.49' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.298 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž\„ïoÂ(:ä ÊÑëÔꩪÍ network.recv‚633170 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.2421875& disk.nvme0c0n1.out‚35062.349609375 disk.sda1.out‚0 disk.dm-0.in‚ 4518.07421875! disk.dm-0.out‚35062.349609375 network.sent‚ 1.79065e+06 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 4518.07421875$ disk.nvme0n1.out‚35062.349609375 disk.sda.in‚0 disk.sda.out‚ 60.5078125 disk.dm-1.out‚ 60.5078125$ memory_percent‚95.19650936126709 disk.sda3.out‚ 60.5078125) disk./.usagePercent‚79.00175745076136# disk.nvme0c0n1.in‚ 4518.07421875 disk.sda3.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 787.00390625# proc.memory.rssMB‚ 2956.67578125* proc.memory.percent‚18.046116828918457$ disk./.usageGB‚1177.183277130127 disk.dm-1.in‚0 cpu‚25.232928434912004 proc.cpu.threads‚17¼•j…’Ä‚(:‡ ÙÑëÔ™ÆÿÊ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚175.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ £Š8Å‚(:ã ÙÑëÔÐê¨Í disk.dm-1.out‚ 60.8828125 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 688.74609375* proc.memory.percent‚18.647003173828125 proc.cpu.threads‚17 disk.sda3.in‚0( disk./.usagePercent‚79.0059626305188$ disk./.usageGB‚1177.245937347412 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 4518.07421875' disk.nvme0c0n1.out‚35111.1552734375$ memory_percent‚95.79622745513916 proc.memory.rssMB‚3055.125 disk.sda3.out‚ 60.8828125 disk.sda.out‚ 60.8828125 disk.sda1.in‚0 cpu‚25.241910297315282! disk.nvme0n1.in‚ 4518.07421875% disk.nvme0n1.out‚35111.1552734375 network.sent‚ 1.841499e+06 network.recv‚637647 disk.drbd1260.out‚ 0.24609375 disk.dm-0.Átˆ%6in‚ 4518.07421875" disk.dm-0.out‚35111.1552734375Ö5ŸùÆ‚(:’ èÑëÔ½¸‡Ë gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚143.448' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.689600000000006 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 j –uÇ‚(:ê èÑëÔߌÇÍ) disk./.usagePercent‚79.00594086987837 disk.sda2.in‚0' disk.nvme0c0n1.out‚35468.8896484375 disk.dm-1.in‚0 disk.dm-1.out‚ 61.41015625$ memory_percent‚95.80018520355225( proc.memory.availableMB‚ 688.09765625 cpu‚25.281398630291882 network.sent‚ 1.847215e+06 network.recv‚641252 disk.sda2.out‚0 disk.drbd1260.out‚ 0.24609375! disk.nvme0n1.in‚ 4518.07421875 disk.sda3.out‚ 61.41015625 disk.sda.out‚ 61.41015625" disk.dm-0.out‚35468.8896484375 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 4518.07421875% disk.nvme0n1.out‚35468.8896484375 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 4518.07421875 proc.memory.rssMB‚ 3055.40625* proc.memory.percent‚18.648719787597656% disk./.usageGB‚1177.2456130981445 disk.sda3.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17ÅÈ‚(:… ÷ÑëÔȱÍ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÝm–_É‚(:Ô ÷ÑëÔ“§ØÍ disk.dm-0.in‚ 4518.07421875" disk.nvme0n1.out‚ 35812.2734375 disk.sda1.out‚0 disk.dm-0.out‚ 35812.2734375 disk.dm-1.out‚ 61.859375 disk.drbd1260.in‚0$ disk.nvme0c0n1.out‚ 35812.2734375 disk.sda3.out‚ 61.859375) proc.memory.percent‚18.65100860595703 cpu‚25.318715589202768 network.sent‚ 1.852675e+06 network.recv‚645622) disk./.usagePercent‚79.00856417908173 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚95.79970836639404( proc.memory.availableMB‚ 688.17578125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.out‚0.25# disk.nvme0c0n1.in‚ 4518.07421875! disk.nvme0n1.in‚ 4518.07421875 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 61.859375 proc.memory.rssMB‚ 3055.78125% disk./.usageGB‚1177.2847023010254 disk.sda1.in‚0ÖN|å“Ê‚(:ˆ †ÒëÔÔÁòÊ gpu.0.gpu‚24.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.387' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œDr-`Ë‚(:Õ †ÒëÔ¡³ÆÍ proc.cpu.threads‚17 network.recv‚648923) disk./.usagePercent‚79.01856383336916% disk./.usageGB‚1177.4337043762207# disk.nvme0c0n1.in‚ 4649.78515625 disk.sda3.in‚0 disk.sda3.out‚ 62.39453125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.25 disk.sda.out‚ 62.39453125 disk.dm-0.in‚ 4649.78515625 disk.dm-0.out‚ 36095.421875 disk.sda2.in‚0" proc.memory.rssMB‚ 2956.4921875 disk.sda.in‚0# disk.nvme0c0n1.out‚ 36095.421875! disk.nvme0n1.in‚ 4649.78515625 disk.sda1.in‚0 disk.dm-1.out‚ 62.39453125$ memory_percent‚95.19309997558594) proc.memory.percent‚18.04499626159668 cpu‚25.355846592892064 network.sent‚ 1.857616e+06! disk.nvme0n1.out‚ 36095.421875 disk.sda2.out‚0$ proc.memory.availableMB‚787.5625´^fu‘Ì‚(:† •ÒëÔ‰©Ì gpu.0.gpu‚42.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚166.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zîÒ#tÍ‚(:é •ÒëÔ€ìäÍ disk.drbd1260.out‚ 0.25390625' disk.nvme0c0n1.out‚36475.9033203125 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚652674% disk./.usageGB‚1177.4331703186035 disk.dm-1.in‚0 disk.dm-1.out‚ 63.0234375 disk.sda2.in‚0( proc.memory.availableMB‚ 697.11328125* proc.memory.percent‚18.596482276916504 cpu‚25.393934255199518! disk.nvme0n1.in‚ 4649.78515625 disk.dm-0.in‚ 4649.78515625# memory_percent‚95.7451581954956# proc.memory.rssMB‚ 3046.84765625# disk.nvme0c0n1.in‚ 4649.78515625% disk.nvme0n1.out‚36475.9033203125 disk.sda3.in‚0 disk.sda3.out‚ 63.0234375 disk.sda.in‚0 disk.sda.out‚ 63.0234375 disk.sda1.out‚0" disk.dm-0.out‚36475.9033203125 network.sent‚ 1.862904e+06) disk./.usagePercent‚79.01852799231438 disk.drbd1260.in‚0²I#jrÎbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP0ÁîfÏba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPy+ZÝlÐbg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JPÂ(ÕwÑbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPþŠ¬ühÒbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JP@A±lÓbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JP¯ôýlÔbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JPxuÕbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPà?(qÖbl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÒ^dËx×bs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPㇹ ]ØbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JPœDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP}2Ð2É- _wandbruntime‚1909  _runtime‚1909îÃŒ#N*ÊÈT QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚102822907600896 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚111951226895872 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚102822907600896 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 682636800 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55735608 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19925 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚101441735092736 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚214 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚99480726269916 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚30552965 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚101441717089792 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22458401 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚15074238 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 _step‚96 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1381873147904 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚99480028166384 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681064172 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 156224000 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14134971392 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚15074236 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684408973 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚25162718 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1448474266112 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1376024998304 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚99480726269916 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚30552965 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚101441717089792 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22458401 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚15074238 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 _step‚96 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1381873147904 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚99480028166384 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681064172 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 156224000 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14134971392 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚15074236 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684408973 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚25162718 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1448474266112 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1376024998304 F–tÝbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPÇDtÏvÞbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPf8×Ødßb_ tîbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP%o÷øfïba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JP×FJmwðbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP>çÓhcñb^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JPÁ®mòbh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JP®A}|vóbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP ·ætôbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP ~\Ôiõbd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JPZSújöbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPø¬KNl÷bg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JPªó›uøbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPsü„rùbm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP©tqÜ[úbV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JPµ|*iûbd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JPhŸ«;eüb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JP&ºÞmtýbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPÞ*‚Ðsþbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP`o*¶tÿbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPžòÉk€bf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JP…E¦¤kbf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JPW˜ôÀo‚bj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JPqõ›Üoƒbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP&&‚ôl„bg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JPQNAr…bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPøË‚.q†bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPÒ›ˆÚl‡bg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPòÆjˆbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JP] 4An‰bi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JP=婵wŠbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPØb‹Ëp‹bk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP*Fƒ=kŒbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JPaÛakbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP ”3wŽbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP˜iŸbb] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JPöú_jrbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPd©Ôg‘bb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPí’»½d’b_ DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPÞaŒ%r¿bm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP½!ÝjÀbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JP¥eÁb` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP™kíÞlÂbg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JPÏv0ÞpÃbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPŽàwÄbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPhL¢hrÅbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPÕÍ@¤kÆbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPôÍw2Ç- _wandbruntime‚1911  _runtime‚1911sâ’x$ÈÒR TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 700639744 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚99531640585984 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684408973 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 _step‚97 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1377194273360 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55739971 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚25178194 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚99530942482452 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1381680361984 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚98154446312624 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 682636800 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55759898 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚102878358998528 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚7389342 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚113456434626560 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 _runtime‚1911.280054733 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚25178192 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681064172 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚101495995999744 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚7389556 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚101495995999744 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚102877658358784 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end7žøæà/allocation.small_pool.allocated‚30581704 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1382362998784 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1381680361984 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚214 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 700639744 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚55759898 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19925 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570îÆÕ\)ÉÖR aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1377194273360 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55739971 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚25178194 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚99530942482452 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1381680361984 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚98154446312624 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 682636800 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55759898 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚102878358998528 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚7389342 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚113456434626560 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 _runtime‚1911.280054733 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚25178192 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189001216 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681064172 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚101495995999744 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚7389556 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚101495995999744 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚102877658358784 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚30581704 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1382362998784 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1381680361984 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚214 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 700639744 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚55759898 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19925 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 700639744 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚99531640585984 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13684408973 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 _step‚97aÜFàÊ‚(:… ¤ÒëÔ„™ÉÊ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚114.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á+cuË‚(:ê ¤ÒëÔ”¯«Ì' disk.nvme0c0n1.out‚36725.4970703125% disk.nvme0n1.out‚36725.4970703125 disk.sda.in‚0 disk.dm-1.in‚0$ memory_percent‚95.74964046478271 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.25390625# disk.nvme0c0n1.in‚ 4649.78515625 disk.sda1.out‚0 disk.dm-0.in‚ 4649.78515625" disk.dm-0.out‚36725.4970703125# proc.memory.rssMB‚ 3046.88671875* proc.memory.percent‚18.596720695495605 network.sent‚ 1.866847e+06 network.recv‚657118) disk./.usagePercent‚79.01840203660751% disk./.usageGB‚1177.4312934875488! disk.nvme0n1.in‚ 4649.78515625 disk.sda1.in‚0 disk.dm-1.out‚ 63.5546875 cpu‚25.432502545531257 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 63.5546875( proc.memory.availableMB‚ 696.37890625 disk.sda3.out‚ 63.5546875 disk.drbd1260.in‚0¼¹¥NÌbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPtòn¾HÍbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPôdŽJÎbE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPùõ9~JÏbE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP,«°ˆNÐbI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPášÏNÑbI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPIoóHÒbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP:nvžNÓbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP: ‹UJÔbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP–k[vHÕbC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPU@jtNÖbI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPæët-N×bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP#ÏQJØbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPüÒi’N bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPƒØmÂN‘ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPbrpŽH’ bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPÃôÕJ“ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP‹ÊêÌJ” bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‡jÝ[J• bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP"2”tJ– bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPñ;0j;— b6 jet/accuracy_1_of_1"trainer/global_step2:JPrB?H˜ bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÅÑH™ bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP!n°Nš bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÿ‘ðÑH› bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPŽ€ÀÏHœ bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPò}ÄJ bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPQhoÏJž bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPåiKNŸ bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPr:sN  bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPæØ7Ð;¡ b6 jet/accuracy_3_of_7"trainer/global_step2:JPPI¦J¢ bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPQª]&H£ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPUœ]H¤ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPQñ¤J¥ bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP‹äºãN¦ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÜo²:J§ bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPbÿ/®H¨ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÆñ¦ZJ© bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPö/RïNª bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPvJÞN« bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP G~(H¬ bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPОOþH­ bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPºÿØíN® bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPìùª”H¯ bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPv,vJ° bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¬æbúJ± bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPP÷`J² bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‡­_øH³ bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPr~ÒöJ´ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP¾ŒêNµ bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPêòuÖJ¶ bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPú]ŒJ· bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP H¦aN¸ bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP‰U—©N¹ bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP…p¤tHº bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPûãh7N» bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP²­J¼ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP”Ù{¬J½ bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPHµ¥J¾ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP'DˆÓN¿ bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPlá×JÀ bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP’ë‹NÁ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPTÒ¯`N bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPñ/ùÏHà bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPpŒËHÄ bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPЦ3HÅ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPˆ€Ó+HÆ bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP —$4JÇ bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPPò@@È b; particle/accuracy_2_of_7"trainer/global_step2:JPÊiñÓJÉ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPo¦NÊ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¿I% NË bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÌu tHÌ bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPõÃþÎJÍ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¬yÔ³JÎ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPôÔ]NÏ bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPµ±—HÐ bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP¾ì¸ÆNÑ bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPxç²)HÒ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPºKµ€JÓ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¡àHÔ bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP¸œÒMHÕ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPBSòNÖ bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPbè„zH× bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPQ²fûHØ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPÁ Ø0JÙ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP œ‚JÚ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPÔB@™HÛ bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPE7AJÜ bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPĵHÝ bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP·z65NÞ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP°QE“Jß bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP±M`ÛJà bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP“,õŸHá bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPu«^|Jâ bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPˆÊ IHã bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP^rjBJä bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP3 îÌNå bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPýD+ Næ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPk~yxJç bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP.½£™Hè bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPÝ@ãÍHé bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÄëNuNê bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPI¬ PNë bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPήq@ì b; particle/accuracy_1_of_4"trainer/global_step2:JPêÜ,¾Jí bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP‰r™Jî bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPX—µNï bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPñŽÐNð bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP¯uÓ@Jñ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP}Ý|ùHò bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP‰è~ßNó bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP,¬ðHô bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPìÝjG;õ b6 jet/accuracy_4_of_6"trainer/global_step2:JPo ñJö bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP:ã¡V;÷ b6 jet/accuracy_3_of_3"trainer/global_step2:JP5ðT@ø b; particle/accuracy_2_of_5"trainer/global_step2:JP³ia)@ù b; particle/accuracy_2_of_6"trainer/global_step2:JP$ç›öNú bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPB4CHû bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP¬(KkNü bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP8L-]Ný bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP˜»L³Jþ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPä®MHÿ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP]àGJ€ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPí?ØN bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPŽ’L‹H‚ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÑšÔJƒ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¼×ìH„ bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÔeæ:H… bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP€°[ÁH† bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPæ_$âJ‡ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP"@ˆ b; particle/accuracy_3_of_6"trainer/global_step2:JP/ܱ;‰ b6 jet/accuracy_1_of_7"trainer/global_step2:JP<}?ZHŠ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP`ºHòJ‹ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¶¡íßHŒ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPËHseN bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP?ã1@Ž b; particle/accuracy_1_of_2"trainer/global_step2:JP„{ÿrH bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP‰£–ôJ bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP†w\OJ‘ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÁÇwN’ bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÓýlH“ bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPHƒgJ” bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP0ùplN• bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP€{üïN– bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPMRòH— bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP®Öt\N˜ bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPp±îÇN™ bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPD$rrHš bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP;´ÑjH› bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPœ†² Nœ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP/!…nJ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPI¬¾Jž bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP#røöNŸ bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP+Iß;N  bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPôN¡ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP8ƒEŒ<¢ b7 particle/specificity"trainer/global_step2:JP/ͬ°@£ b; particle/accuracy_1_of_7"trainer/global_step2:JP7Á}]J¤ bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPhÍäH¥ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPÓ ¬H¦ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP~†]J§ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP¼¿xN¨ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPŸ&k’N© bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP¿ð†Nª bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPš¶++J« bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPDž4J¬ bE "Purity/4FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP ã7SH­ bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPö8ÈXN® bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÚuyN¯ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPk¯{J° bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPa,H± bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPºªØr;² b6 jet/accuracy_2_of_4"trainer/global_step2:JP½ ³J³ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP08zH´ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÑX ÑNµ bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP æ SN¶ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP8ÐáJ· bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP±q—“J¸ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPbí^H¹ bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÚdrSJº bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPw„J» bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPnm'H¼ bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPœÈFH½ bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP‚,ÿÉG¾ bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:J¾OXbP¡ÿ·¾H¿ bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP=NxöNÀ bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP&¬ÿ-HÁ bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPI°“€J bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¿'_Hà bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP8Á#JÄ bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPgHäv@Å b; particle/accuracy_5_of_6"trainer/global_step2:JP/-NÆ bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPíî(…NÇ bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPð„çYJÈ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPêÐYJÉ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPŽ\Žœ@Ê b; particle/accuracy_4_of_7"trainer/global_step2:JP(GrJË bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPEäõHÌ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPŒJ….JÍ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPVÅvNÎ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP1ÝsHÏ bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP2KÔÁNÐ bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPæïàHÑ bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP£ÑrUJÒ bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÖg)*JÓ bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP¢˜ÝVJÔ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPCfĈJÕ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPbv°HÖ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÐöÌ•J× bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¿ßpXHØ bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP=ì~)NÙ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPþ}ôNÚ bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP` ðDNÛ bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP)ÖNÜ bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPyPe9Ý b4 particle/accuracy"trainer/global_step2:JPDÚ@@Þ b; particle/accuracy_5_of_7"trainer/global_step2:JPTÀDJß bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP’ ŠNà bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP‚xüÍNá bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP0Aw Jâ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP|h¾Nã bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP!9!¸Jä bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÊS‘Jå bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPôhͺ;æ b6 jet/accuracy_2_of_6"trainer/global_step2:JP6WHç bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP–¨z™Nè bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¹OHé bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPP²Q÷Jê bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP¤ÅžNë bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP{§µNì bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPËêÐ@í b; particle/accuracy_3_of_7"trainer/global_step2:JP±'9eJî bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPo1ÃHï bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPl²OpJð bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPjë ÒNñ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPOØNò bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÏÆŠNó bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP ä¡*Jô bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÚ®&MHõ bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP³ý”Hö bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPçÖjH÷ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP½F†"Nø bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP¦¸~€Hù bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP¢cp¿Jú bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÝ?¦oJû bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP<•õÄJü bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP­ãðqJý bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPy÷hHþ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÇí}Nÿ bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPã ùH€ bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP:ˆRXJ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPh–@nN‚ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPüÑ#;ƒ b6 jet/accuracy_1_of_6"trainer/global_step2:JPb{×&J„ bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPÔu1H… bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP‘â)J† bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPǼNN‡ bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPŸ8ø¡@ˆ b; particle/accuracy_5_of_5"trainer/global_step2:JPà ¢ŽH‰ bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPP ëßNŠ bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÞâ€J‹ bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP%¿Ï#JŒ bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP@CûÕJ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPßV ®JŽ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPYÓN bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPK¶TH bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP ‰r J‘ bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPöf—Û;’ b6 jet/accuracy_5_of_7"trainer/global_step2:JPøžm›H“ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP0'¾^J” bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPeØ#ýN• bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP•‡lQH– bC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPªG­TH— bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPQ¸{½J˜ bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP½T[‹J™ bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP2±¹Nš bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP…¾J¨H› bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP-úQÞNœ bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÊšEï@ b; particle/accuracy_1_of_6"trainer/global_step2:JPg÷÷Jž bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP›v)£HŸ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÍÈßmH  bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP>\°N¡ bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP+~QJ¢ bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP´au_;£ b6 jet/accuracy_1_of_2"trainer/global_step2:JPÔ¼…J¤ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP@¯56H¥ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP$´_H¦ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPW[ôJ§ bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP0¢ßßH¨ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP†H© bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPVSµüHª bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP:›ôÈH« bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP”n\VJ¬ bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPc©N­ bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPtŠ¹ N® bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÚ ‹äN¯ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP*üY>@° b; particle/accuracy_4_of_4"trainer/global_step2:JPUú4VJ± bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPvYÔJ² bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPN¸ bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPB·£×N¹ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÎäœèNº bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPSª’ó@» b; particle/accuracy_6_of_6"trainer/global_step2:JP3LŠ%J¼ bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPcîÑJ½ bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP<ÇÇN¾ bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP#g…_J¿ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPUÜJÀ bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPTÌ·×JÁ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPqIÅæN bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP,òb(Nà bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP|"ÍøJÄ bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP™ZhHÅ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPœ7CJÆ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP\’ËRNÇ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÈõÊM;È b6 jet/accuracy_5_of_5"trainer/global_step2:JPÃ)Œ@É b; particle/accuracy_2_of_2"trainer/global_step2:JPXî¾JÊ bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPÓjC}HË bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPq 0NÌ bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP °SÅJÍ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPIêÁ;Î b6 jet/accuracy_2_of_3"trainer/global_step2:JPì’+HÏ bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPó,bJÐ bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP_I8ûJÑ bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP8€b¨JÒ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP ·‰âJÓ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP\ã(µJÔ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPKJ|´HÕ bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP2E²¦JÖ bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPlyN× bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPaVJf@Ø b; particle/accuracy_1_of_5"trainer/global_step2:JPåžJÙ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPU {HÚ bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPðì«HÛ bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPz4HÜ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPñ,ímJÝ bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP b—NÞ bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP+šünHß bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPœ™3!Hà bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP[âHá bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP\ѵ¥Hâ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP–½ÑHã bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÙË;¼Nä bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP`¿LXJå bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‘à|Jæ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP}¯ÂÙNç bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÑ\ÏNè bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP.ñ8Jé bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPóö.Jê bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPœr/Hë bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP­ÌÞ¼Jì bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP]þ‰²Hí bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPK=Jî bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPmlH]Nï bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPt‹¸Jð bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPjÆq¶Jñ bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPó³ÚNò bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPؾQßNó bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP—÷ Ú;ô b6 jet/accuracy_1_of_4"trainer/global_step2:JP³•TJõ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP±T¸Jö bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPâyé½N÷ bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP|„Jø bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPk3²@ù b; particle/accuracy_1_of_1"trainer/global_step2:JP3çøJú bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP±™ÇJû bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPëhó—Nü bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPæõASNý bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPH?ô Jþ bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÔ'@SJÿ bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP_Ñ3€J€ bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP³[ŃH bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP[@}H‚ bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPfòƒ0Jƒ bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPyFH©J„ bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPvjEN… bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP‰£çH† bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¹Í0‡H‡ bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPèM’Jˆ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP:òýN‰ bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP!¹hÅNŠ bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP¦Š3©H‹ bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP£aJŒ bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPkkаJ bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPËl?—JŽ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPSy°?N bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP›ÐíH bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÜ=qÅH‘ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP\ÀH’ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPú¢’J“ bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP9««J” bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÒMß*N• bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP•v¼ý;– b6 jet/accuracy_7_of_7"trainer/global_step2:JPµ¢J— bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¶g‡`J˜ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP®îBH™ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP©U9œHš bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP»3¿PJ› bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPw:¬¹Nœ bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPP¥GLN bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPáAn´Hž bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPN9´ÕJŸ bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPrüXIH  bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPtü N¡ bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPpN7ÏJ¢ bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¢ÇJ£ bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP´„ÕH¤ bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP3µ\ N¥ bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPO ¾H¦ bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPvñ@H§ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPË b‹N¨ bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP9ïΠH© bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP‡£}rJª bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP|K´N« bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPó*ãÂ@¬ b; particle/accuracy_3_of_4"trainer/global_step2:JP#]´H­ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPG+ïÀH® bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP„nãN¯ bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP’SázH° bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP‹9CJ± bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP»pTN² bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPˆý J³ bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPàñv 8´ b3 particle/f_score"trainer/global_step2:JP…ÞÄàHµ bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPìGó@H¶ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP™±DJ· bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP[ËWN¸ bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP“{ϯ;¹ b6 jet/accuracy_3_of_5"trainer/global_step2:JP¡ýcÅNº bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPl¿Ö¤N» bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÑÄÚJ¼ bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP"WyôN½ bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPq mí;¾ b6 jet/accuracy_3_of_6"trainer/global_step2:JPíWï¡N¿ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPö{oJÀ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP >áJÁ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP5ͬ;N bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPm$#7Nà bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPòÚZNÄ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÕEo²JÅ bE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¸™HÆ bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP ½ -JÇ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPËÄNÈ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPºáHÉ bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP,o‘”HÊ bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPLêZMJË bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPá}•NHÌ bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÖ’:‡JÍ bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP7A~HÎ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP¨ì2JÏ bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP7NÐ bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP7Æb@Ñ b; particle/accuracy_2_of_3"trainer/global_step2:JPõÍ©’JÒ bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPà®Ç‚HÓ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÖË’NÔ bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPHš,^JÕ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPnÖ‰JÖ bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP–ÙB:J× bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPãHØ bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP0§JÙ bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPûcMŽ;Ú b6 jet/accuracy_4_of_4"trainer/global_step2:JPö¤ ðHÛ bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP™Ö³µNÜ bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÃk3NÝ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPDpYJÞ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPƒ€cNß bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPx¸’FJà bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP5ud™Ná bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP(1kJâ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPíB|Nã bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP‚™ãGä bB validation_average_jet_accuracy"trainer/global_step2:JPOe ˆJå bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPTfHæ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP-Å©INç bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP 6RCJè bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP0ÜÖJé bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPøƒNê bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP|Í" ;ë b6 jet/accuracy_1_of_3"trainer/global_step2:JPMªœSHì bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPªzÔNí bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPc‹M›Jî bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÜá¢è<ï b7 particle/sensitivity"trainer/global_step2:JP"é=Hð bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPw©üJñ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPê;ÜJò bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP—ºÏjHó bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP“S>uNô bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP"+~¬Jõ bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPñ§´aNö bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP‡‰¶N÷ bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP²|$R;ø b6 jet/accuracy_2_of_2"trainer/global_step2:JPã`‡Jù bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPR“¸xHú bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÂ…ÝöHû bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP´'eÓHü bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPV}†Ný bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPšCq Jþ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPS~Ì·;ÿ b6 jet/accuracy_4_of_7"trainer/global_step2:JP œ\Ÿ@€ b; particle/accuracy_7_of_7"trainer/global_step2:JP–Ñ4J bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÉà];‚ b6 loss/val_total_loss"trainer/global_step2:JPŒOP1Jƒ bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP…XVN„ bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÉÙŠõN… bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPz;NÇH† bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP"ÙÐN‡ bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPS_˜u;ˆ b6 jet/accuracy_5_of_6"trainer/global_step2:JP·+hJ‰ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPX ƒŸHŠ bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPp¾µ±J‹ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPL#ô½JŒ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPQó(þJ bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¶J}£HŽ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP&ø[ÃN bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPïLÀ—N bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÜ»&ÍN‘ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP~'§ÀN’ bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPD˜ŠH“ bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP[ˆá H” bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPìræµH• bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP` O.N– bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPëìN— bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP² ž1N˜ bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP «iJ™ bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÜã9Jš bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP~—[N› bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPO[ʺJœ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPt·ã‡N bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPN$žÖNž bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP´®’JŸ bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP"žëtH  bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPTöBH¡ bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPDoWH¢ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP1¾ÄýH£ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP¹G0J¤ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPªmöãH¥ bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP ¸#H¦ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP"02 N§ bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPç‡ÏÜN¨ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP,§©J© bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÝè» Nª bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÒ˜ÒN« bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPaa¦N¬ bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPrÎN­ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP@+¸J® bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÁ×4…J¯ bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPž¸W‘J° bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPH¬ºH± bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPo#N©H² bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPP9Ü^J³ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPã´‚*H´ bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÄŠWJNµ bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP7˜» @¶ b; particle/accuracy_4_of_5"trainer/global_step2:JP‘(w+;· b6 jet/accuracy_6_of_6"trainer/global_step2:JPå †ãH¸ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPžÈËN¹ bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP ³ßJº bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPƼ0ƒJ» bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP·ûFH¼ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPÚС;½ b6 jet/accuracy_6_of_7"trainer/global_step2:JP¥©{~N¾ bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP‹®~ZJ¿ bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPVž±JÀ bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPì- NÁ bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP,îAJ bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP(ZÖŒHà bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP*;]NÄ bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPpØ­DNÅ bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÕ NÆ bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP2&¢{JÇ bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÒuË­@È b; particle/accuracy_4_of_6"trainer/global_step2:JP.ãüHÉ bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÄ‘pð;Ê b6 jet/accuracy_2_of_7"trainer/global_step2:JP"¡fNË bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP(¸ªíNÌ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÛù¬}NÍ bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP+ð¨óNÎ bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP»Vp$HÏ bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÊT-ñJÐ bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPãc(íJÑ bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÒ6Ý9NÒ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPC[yˆHÓ bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPA*Ì­NÔ bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP6 ;Õ b6 jet/accuracy_1_of_5"trainer/global_step2:JPé‹‹ Ö b6 jet/accuracy_4_of÷>~Ë#_5"trainer/global_step2:JPiJåH× bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP{*YJØ bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP£XšJÙ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÆMD·JÚ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPþE~HÛ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP•HÜ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÜÏ“ÈNÝ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP4ÑJJÞ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPꎛŠHß bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPfL>Nà bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPiéç¨Há bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPä%ÏHâ bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÞaKJã bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPGOS+Hä bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPQc«-Nå bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPåpÐJæ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP΋ïJç bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPz (%Nè bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP3œð×Né bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP±¯0Jê bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÁ8D™Jë bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¯à ¤Nì bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP'02Ní bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPVˆ¿Í;î b6 jet/accuracy_3_of_4"trainer/global_step2:JPÏÑ¡Hï bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPõ¦¦óJð bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP_‘´Nñ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÊú!„Hò bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPh™šœHó bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPzÊëHô bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPQ2#1Nõ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPƒ;ÜèHö bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP]¨ æN÷ bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPy>©@ø b; particle/accuracy_1_of_3"trainer/global_step2:JP$û7ÒHù bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP©€§£Nú bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPN¿æJû bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP}]ÌœHü bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP)v´Hý bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPr—ºPJþ bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPVÈá%Hÿ bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP2NŸ}J€bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP+Ù²¥JbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP_çÿ™H‚bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPŒngHƒbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP`ké{N„bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP×ê'{H…bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP‰/(J†bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÌ‹Ù'J‡bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPŠA3ˆHˆbC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPö˜–DN‰bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPB¾«NŠbI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP†üi—N‹bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPBýFZJŒbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP`‡Û2HbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP)È©NŽbI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPš×`ÖHbC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPO¦] JbE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP?·1J‘bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP¼ˆ®ðJ’bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP;°<*@“b; particle/accuracy_6_of_7"trainer/global_step2:JPù;'ýJ”bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPb6ŠùN•bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÉÄ H–bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÛåôJ—bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPè:†^N˜bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPó·N™bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÎá„+NšbI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPkfýM2›- _wandbruntime‚1928  _runtime‚1928O³cújœÔ¦ :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.41815832257270813 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.28070175647735596 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24069271981716156 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.09375 particle/accuracy_4_of_6‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_2_of_7‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929622411727905 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.027327196672558784 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39629629254341125 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.8337429761886597 ,jet/accuracy_4_of_5‚0.004320276435464621 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.18518517911434174 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1666666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9901390075683594 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.19881467521190643 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9984127283096313 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.22351820766925812 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983747601509094 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.16119860112667084 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.16358280181884766 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.08011901378631592 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.055443357676267624 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.43065476417541504 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17677566409111023 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004557381384074688 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9890351295471191 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40297624468803406 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0012282690731808543 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35740092396736145 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22747653722763062 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2084435671567917 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0017857142956927419 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24554337561130524 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.23064185678958893 particle/accuracy_6_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889702796936035 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5227272510528564 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20717909932136536 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3523421287536621 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.38555118441581726 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2070503979921341 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34779876470565796 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9757651686668396 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1712646186351776 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.13341128826141357 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 6 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.699999988079071 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 % _timestamp‚1.7885371328046453e+09 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6198697686195374 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.29281607270240784 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22897560894489288 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.3169611096382141 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2083333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9909798502922058 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3652365803718567 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.010267069563269615 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.33914726972579956 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.231037437915802 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11170835047960281 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3226473033428192 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2694927453994751 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.12695449590682983 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.16358280181884766 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 jet/accuracy_3_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4072107672691345 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2064051330089569 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547619104385376 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22165541350841522 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.31705617904663086 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.14318858087062836 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.15510118007659912 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9942857027053833 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.16668319702148438 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.30858588218688965 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1584213525056839 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1462971717119217 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1642438769340515 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.28569796681404114 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11961700767278671 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2864942252635956 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.1304597556591034 particle/accuracy_2_of_7‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1786254346370697 :"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.49242421984672546 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3078828752040863 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9831178188323975 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.47930651903152466 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3594253361225128 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.22223693132400513 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18424014747142792 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3922285735607147 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3527986407279968 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2950292229652405 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.02070624940097332 particle/accuracy_2_of_5‚1 particle/accuracy_2_of_6‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991640448570251 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3569188416004181 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882314205169678 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011010902933776379 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.1526016741991043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.25327223539352417 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 particle/accuracy_3_of_6‚1 jet/accuracy_1_of_7‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22647908329963684 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.34539473056793213 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33703702688217163 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.47626087069511414 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01267281174659729 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.347222238779068 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895581007003784 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *particle/specificity‚0.937961220741272 particle/accuracy_1_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.0833333358168602 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24980640411376953 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0267857164144516 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1872428059577942 +jet/accuracy_2_of_4‚0.29403799772262573 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3879166543483734 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15148913860321045 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4447654187679291 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14139993488788605 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9930911660194397 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4107142984867096 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.13420480489730835 particle/accuracy_4_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9750000238418579 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.14198914170265198 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23569777607917786 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4047071933746338 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8311681747436523 particle/accuracy_5_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.40668246150016785 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3247126340866089 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.6089743375778198 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856758117675781 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11099059134721756 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_7‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9034392237663269 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.15515285730361938 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.1508481651544571 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3119047284126282 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3853534162044525 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9876178503036499 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 )jet/accuracy_1_of_6‚0.942307710647583 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31953325867652893 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3472222685813904 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1955205500125885 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.347222238779068 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09504064172506332 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3452411890029907 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9787889719009399 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2451566457748413 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.5701718926429749 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.40666666626930237 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9914125204086304 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4043179750442505 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3519892990589142 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17886947095394135 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.22413793206214905 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4771568477153778 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.21555963158607483 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39201390743255615 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.11481481045484543 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_2‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 trainer/global_step‚1208 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 +jet/accuracy_2_of_3‚0.19284166395664215 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.22784049808979034 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06993953138589859 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9881049990653992 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16607771813869476 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.380952388048172 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9915846586227417 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0103374645113945 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22501903772354126 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5176282525062561 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.43565765023231506 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3069276809692383 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17879605293273926 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.23333331942558289 epoch‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.40537768602371216 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018280884250998497 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8041594624519348 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt3SRqqt0FBt/SRqqt_purityî•·H)‚0.12241212278604507 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.14043210446834564 particle/accuracy_1_of_1‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2083333432674408 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3958333432674408 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9889071583747864 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.086221843957901 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004515261854976416 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977920651435852 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9841363430023193 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.14763414859771729 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23055556416511536 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1940559297800064 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9945833086967468 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8975694179534912 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 _runtime‚1928.118872931 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19592782855033875 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1666666865348816 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1696295142173767 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1308279037475586 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998443067073822 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9825581312179565 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.345049113035202 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.21562859416007996 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.12222222238779068 'particle/f_score‚0.3195931911468506 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3958333432674408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6819066405296326 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,jet/accuracy_3_of_5‚0.051350485533475876 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 +jet/accuracy_3_of_6‚0.11538461595773697 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04812469705939293 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13956664502620697 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.18911372125148773 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14034704864025116 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.13511043787002563 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22530865669250488 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.0902777761220932 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13753752410411835 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_3‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10116483271121979 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25509122014045715 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007244881708174944 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3561508059501648 ,jet/accuracy_4_of_4‚0.005474243313074112 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.014367816969752312 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3171350061893463 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8995273113250732 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07231307774782181 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5138888955116272 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.6911481618881226 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9830790162086487 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 +particle/sensitivity‚0.2541307806968689 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16763435304164886 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.1681034415960312 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.08311477303504944 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.218540221452713 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 _step‚98 *loss/val_total_loss‚51.214681993508265 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8100711703300476 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2807539403438568 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_5_of_6‚0.01923076994717121 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11099059134721756 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.224203959107399 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.43665847182273865 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0008540000999346375 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8874268531799316 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07040458172559738 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9833365678787231 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9991645812988281 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 <"Purity/0FRt2SRqqt0FBt/event_purity‚0.0023767082020640373 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.26620373129844666 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.699999988079071 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22220948338508606 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 jet/accuracy_6_of_6‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9991645812988281 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.20600225031375885 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 jet/accuracy_6_of_7‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656¤Ø©ªVئ 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.47930651903152466 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3594253361225128 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.22223693132400513 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18424014747142792 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_4‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3922285735607147 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3527986407279968 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2950292229652405 +jet/accuracy_4_of_6‚0.01923076994717121 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.02070624940097332 particle/accuracy_2_of_5‚1 particle/accuracy_2_of_6‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9991640448570251 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3569188416004181 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882314205169678 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011010902933776379 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.1526016741991043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.25327223539352417 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 particle/accuracy_3_of_6‚1 jet/accuracy_1_of_7‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22647908329963684 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.34539473056793213 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33703702688217163 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.47626087069511414 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01267281174659729 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.347222238779068 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895581007003784 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.3571428656578064 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 *particle/specificity‚0.937961220741272 particle/accuracy_1_of_7‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.0833333358168602 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24980640411376953 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0267857164144516 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1872428059577942 +jet/accuracy_2_of_4‚0.29403799772262573 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3879166543483734 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15148913860321045 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4447654187679291 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14139993488788605 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9930911660194397 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4107142984867096 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.13420480489730835 particle/accuracy_4_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9750000238418579 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.14198914170265198 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23569777607917786 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4047071933746338 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8311681747436523 particle/accuracy_5_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.40668246150016785 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3247126340866089 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.6089743375778198 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856758117675781 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11099059134721756 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_7‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9034392237663269 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.15515285730361938 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.1508481651544571 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3119047284126282 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3853534162044525 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9876178503036499 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.0714285746216774 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 )jet/accuracy_1_of_6‚0.942307710647583 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31953325867652893 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3472222685813904 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475178182125092 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1955205500125885 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_7‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.347222238779068 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09504064172506332 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3452411890029907 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9787889719009399 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2451566457748413 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.5701718926429749 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.40666666626930237 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9914125204086304 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4043179750442505 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3519892990589142 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17886947095394135 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.22413793206214905 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4771568477153778 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.21555963158607483 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39201390743255615 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.11481481045484543 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_2‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 trainer/global_step‚1208 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 +jet/accuracy_2_of_3‚0.19284166395664215 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.22784049808979034 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.06993953138589859 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9881049990653992 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16607771813869476 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.380952388048172 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9915846586227417 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0103374645113945 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22501903772354126 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5176282525062561 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.43565765023231506 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3069276809692383 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17879605293273926 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.23333331942558289 epoch‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.40537768602371216 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018280884250998497 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8041594624519348 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.12241212278604507 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.14043210446834564 particle/accuracy_1_of_1‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2083333432674408 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3958333432674408 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9889071583747864 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.086221843957901 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004515261854976416 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977920651435852 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9841363430023193 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.14763414859771729 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23055556416511536 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1940559297800064 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9945833086967468 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8975694179534912 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 _runtime‚1928.118872931 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19592782855033875 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1666666865348816 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1696295142173767 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1308279037475586 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998443067073822 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9825581312179565 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.345049113035202 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.21562859416007996 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.12222222238779068 'particle/f_score‚0.3195931911468506 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3958333432674408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6819066405296326 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,jet/accuracy_3_of_5‚0.051350485533475876 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 +jet/accuracy_3_of_6‚0.11538461595773697 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04812469705939293 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13956664502620697 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.18911372125148773 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14034704864025116 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.13511043787002563 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22530865669250488 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.0902777761220932 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13753752410411835 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 particle/accuracy_2_of_3‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10116483271121979 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25509122014045715 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007244881708174944 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3561508059501648 ,jet/accuracy_4_of_4‚0.005474243313074112 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.014367816969752312 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3171350061893463 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8995273113250732 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07231307774782181 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5138888955116272 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.6911481618881226 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9830790162086487 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 +particle/sensitivity‚0.2541307806968689 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16763435304164886 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.1681034415960312 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.08311477303504944 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.218540221452713 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 particle/accuracy_7_‹­zˆµ<of_7‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 _step‚98 *loss/val_total_loss‚51.214681993508265 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8100711703300476 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2807539403438568 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_5_of_6‚0.01923076994717121 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11099059134721756 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.224203959107399 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.43665847182273865 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0008540000999346375 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8874268531799316 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07040458172559738 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9833365678787231 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9991645812988281 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 <"Purity/0FRt2SRqqt0FBt/event_purity‚0.0023767082020640373 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.26620373129844666 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.699999988079071 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22220948338508606 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 jet/accuracy_6_of_6‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9991645812988281 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.20600225031375885 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 jet/accuracy_6_of_7‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.41815832257270813 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.28070175647735596 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24069271981716156 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.09375 particle/accuracy_4_of_6‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 jet/accuracy_2_of_7‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929622411727905 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.027327196672558784 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39629629254341125 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.8337429761886597 ,jet/accuracy_4_of_5‚0.004320276435464621 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.18518517911434174 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.1666666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9901390075683594 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.19881467521190643 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9984127283096313 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.22351820766925812 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983747601509094 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.16119860112667084 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.16358280181884766 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.08011901378631592 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.055443357676267624 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.43065476417541504 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17677566409111023 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.004557381384074688 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9890351295471191 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40297624468803406 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0012282690731808543 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35740092396736145 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22747653722763062 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2084435671567917 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0017857142956927419 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24554337561130524 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.23064185678958893 particle/accuracy_6_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9889702796936035 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5227272510528564 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20717909932136536 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3523421287536621 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.38555118441581726 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2070503979921341 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34779876470565796 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9757651686668396 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1712646186351776 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.13341128826141357 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 6 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.699999988079071 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 % _timestamp‚1.7885371328046453e+09 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6198697686195374 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.29281607270240784 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22897560894489288 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.3169611096382141 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2083333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9909798502922058 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3652365803718567 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.010267069563269615 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.33914726972579956 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.231037437915802 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.11170835047960281 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3226473033428192 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2694927453994751 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.12695449590682983 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.16358280181884766 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 jet/accuracy_3_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4072107672691345 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2064051330089569 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547619104385376 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22165541350841522 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.31705617904663086 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.14318858087062836 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.15510118007659912 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9942857027053833 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.16668319702148438 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.30858588218688965 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.1584213525056839 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.1462971717119217 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1642438769340515 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.28569796681404114 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11961700767278671 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2864942252635956 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.1304597556591034 particle/accuracy_2_of_7‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.1786254346370697 :"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.49242421984672546 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3078828752040863 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9831178188323975 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367b²Vhž‚(:„ ³ÒëÔ¨Ò«Î gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q÷ÞsŸ‚(:è ³ÒëÔªŠÛÐ disk.dm-1.in‚0 disk.sda2.out‚0 network.recv‚663195) disk./.usagePercent‚79.07820898071152# disk.nvme0c0n1.in‚ 4649.78515625! disk.nvme0n1.in‚ 4649.78515625( proc.memory.availableMB‚ 661.55078125# proc.memory.rssMB‚ 3718.46484375) proc.memory.percent‚22.69570827484131 proc.cpu.threads‚17 disk.drbd1260.in‚0& disk.nvme0n1.out‚36899.69384765625 disk.sda.out‚ 64.390625 disk.dm-0.in‚ 4649.78515625$ memory_percent‚95.96221446990967 cpu‚25.450291011963895 disk.drbd1260.out‚ 0.2578125 disk.sda3.out‚ 64.390625 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 64.390625 disk.sda2.in‚0 network.sent‚ 1.937854e+06% disk./.usageGB‚1178.3224601745605( disk.nvme0c0n1.out‚36899.69384765625 disk.sda3.in‚0 disk.sda1.out‚0# disk.dm-0.out‚36899.69384765625‚ «º“ ‚(:ˆ ÂÒëÔœ†ìÊ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚165.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»¼&y¡‚(:î ÂÒëÔ–ÿÆÍ disk.sda2.in‚0 disk.drbd1260.out‚ 0.2578125! disk.nvme0n1.in‚ 4784.83203125) proc.memory.availableMB‚ 1324.44140625 proc.cpu.threads‚17% disk./.usageGB‚1178.9424667358398 disk.sda.out‚ 64.85546875# disk.dm-0.out‚37888.90087890625 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 4784.83203125) disk./.usagePercent‚79.11981814125915 disk.sda3.out‚ 64.85546875 disk.sda.in‚0( disk.nvme0c0n1.out‚37888.90087890625$ memory_percent‚91.91625118255615# proc.memory.rssMB‚ 3057.23828125) proc.memory.percent‚18.65990161895752 cpu‚25.44799586725956 network.recv‚668302 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 4784.83203125 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚37888.90087890625 network.sent‚ 2.063139e+06 disk.sda3.in‚0 disk.dm-1.out‚ 64.855468752KáÀy¢‚(:î ÑÒëÔ¢žôÍ( proc.memory.availableMB‚ 1324.4296875 cpu‚25.445766386789664 disk.sda2.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 4784.83203125) proc.memory.percent‚18.65990161895752 proc.cpu.threads‚17 network.sent‚ 2.066391e+06$ disk./.usageGB‚1178.939868927002 disk.dm-1.in‚0 disk.dm-1.out‚ 65.68359375 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 4784.83203125$ memory_percent‚91.91632270812988 network.recv‚671958 disk.drbd1260.out‚ 0.26171875( disk.nvme0c0n1.out‚39035.22802734375! disk.nvme0n1.in‚ 4784.83203125& disk.nvme0n1.out‚39035.22802734375 disk.sda.out‚ 65.68359375 disk.sda1.in‚0# proc.memory.rssMB‚ 3057.23828125) disk./.usagePercent‚79.11964380012833 disk.sda2.out‚0 disk.sda3.out‚ 65.68359375 disk.sda1.out‚0# disk.dm-0.out‚39035.22802734375ànŸ˜’£‚(:‡ ÑÒëÔƒâïÎ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.293 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìW¸Þ“¤‚(:ˆ àÒëÔ¾ÛÊÌ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.159' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8318 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ËJKm¥‚(:â àÒëÔ¨œÐ network.recv‚676415 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 1323.9296875 cpu‚25.443394920808093$ disk./.usageGB‚1178.963279724121 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 4784.83203125& disk.nvme0n1.out‚39246.23583984375 disk.sda.in‚0 disk.dm-1.out‚66.1875# proc.memory.rssMB‚ 3057.23828125) disk./.usagePercent‚79.12121491836595 disk.drbd1260.out‚ 0.26171875# disk.nvme0c0n1.in‚ 4784.83203125( disk.nvme0c0n1.out‚39246.23583984375 disk.sda3.out‚66.1875 disk.sda.out‚66.1875 disk.dm-0.in‚ 4784.83203125# disk.dm-0.out‚39246.23583984375 network.sent‚ 2.072345e+06 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚91.91937446594238) proc.memory.percent‚18.65990161895752 proc.cpu.threads‚17~¹h2¦- _wandbruntime‚1989  _runtime‚1989?GçƧÀ trainer/global_step‚1249 % _timestamp‚1.7885371941133733e+09 _runtime‚1989.424455398 _step‚99 & lr-AdamW/pg1‚0.0016854364942677543 & lr-AdamW/pg2‚0.0016854364942677543‰,²v欀 _runtime‚1989.424455398 _step‚99 & lr-AdamW/pg1‚0.0016854364942677543 & lr-AdamW/pg2‚0.0016854364942677543 trainer/global_step‚1249 % _timestamp‚1.7885371941133733e+09c›·4Ó2©- _wandbruntime‚1989  _runtime‚1989yf§ì'ªæO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106899757519872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105466151223808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _step‚100 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7691014 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1428246634573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105466133220864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106899757519872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117850339296768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1502109686784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103398011081993 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57855045 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26185780 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31695607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚1249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106898829546496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1432696325632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1433606296064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885371941153743e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31695607 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57881389 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23381720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15690706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117850086573568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 _runtime‚1989.426842808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31669265 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26185780 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31669265 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103398936096349 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26185782 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1502338469888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23381107 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1433606296064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106898829546496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116347976886784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105466133220864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚106899757519872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117850339296768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.smal¤ù#l_pool.freed‚ 1502109686784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103398011081993 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57855045 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26185780 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31695607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚1249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106898829546496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1432696325632 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1433606296064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885371941153743e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31695607 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57881389 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23381720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15690706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117850086573568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 _runtime‚1989.426842808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31669265 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26185780 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31669265 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103398936096349 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26185782 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1502338469888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23381107 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1433606296064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106898829546496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116347976886784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 _step‚100 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7691014 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0dˆÉ> 2¬- _wandbruntime‚1990  _runtime‚1990IÀ¯ö&­ðM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57900036 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚1249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105547752306688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26205942 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117941453885440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1428455932855 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26205942 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57926406 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105547770309632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57926406 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106982498079232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31694096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23398890 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117941719178240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105547752306688 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105547770309632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚101 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106981570092544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23399496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31720464Co—ú&®ôM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105547770309632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57926406 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106982498079232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31694096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23398890 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117941719178240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105547752306688 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105547770309632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚101 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106981570092544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23399496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31720464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57900036 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚1249 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105547752306688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26205942 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117941453885440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1428455932855 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26205942 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57926406 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712eü¾§A2¯- _wandbruntime‚1990  _runtime‚1990×e"JT°Î _runtime‚1990.643385956 &loss/total_loss‚52.356274932776614 1loss/SRqqt1/detection_loss‚2.7528498470783234 2loss/SRqqt1/assignment_loss‚2.7355887431364794 2loss/SRqqt2/assignment_loss‚2.4493702738122507 .loss/FRt4/detection_loss‚1.622979203718645 1loss/FBt2/assignment_loss‚0.22647523591964602 /loss/FRt2/detection_loss‚2.5613212044749942 trainer/global_step‚1249 % _timestamp‚1.7885371953325717e+09 2loss/SRqqt4/assignment_loss‚1.2895909976365605 /loss/FBt1/assignment_loss‚0.529690071552371 /loss/FRt1/detection_loss‚2.7813482121987776 .loss/FRt3/detection_loss‚2.828950728240766 /loss/SRqqt3/detection_loss‚2.11375892162323 /loss/FBt2/detection_loss‚0.9934644727189033 /loss/FBt4/detection_loss‚3.3055553793907166 0loss/FRt4/assignment_loss‚2.1014610155304867 1loss/SRqqt3/assignment_loss‚2.087014602763312 1loss/SRqqt4/detection_loss‚1.0790759134825652 _step‚102 /loss/SRqqt2/detection_loss‚2.61641126871109 -loss/FBt1/detection_loss‚2.63222203652064 .loss/FBt3/detection_loss‚3.069415472447872 epoch‚1 /loss/FRt1/assignment_loss‚4.525960553776134 .loss/FRt2/assignment_loss‚3.66101678950446 /loss/FRt3/assignment_loss‚4.103075115304244 1loss/FBt3/assignment_loss‚0.26605815763650514 1loss/FBt4/assignment_loss‚0.02362071559764445µíé¸X±Ò /loss/SRqqt2/detection_loss‚2.61641126871109 -loss/FBt1/detection_loss‚2.63222203652064 .loss/FBt3/detection_loss‚3.069415472447872 epoch‚1 /loss/FRt1/assignment_loss‚4.525960553776134 .loss/FRt2/assignment_loss‚3.66101678950446 /loss/FRt3/assignment_loss‚4.103075115304244 1loss/FBt3/assignment_loss‚0.26605815763650514 1loss/FBt4/assignment_loss‚0.02362071559764445 _runtime‚1990.643385956 &loss/total_loss‚52.356274932776614 1loss/SRqqt1/detection_loss‚2.7528498470783234 2loss/SRqqt1/assignment_loss‚2.7355887431364794 2loss/SRqqt2/assignment_loss‚2.4493702738122507 .loss/FRt4/detection_loss‚1.622979203718645 1loss/FBt2/assignment_loss‚0.22647523591964602 /loss/FRt2/detection_loss‚2.5613212044749942 trainer/global_step‚1249 % _timestamp‚1.7885371953325717e+09 2loss/SRqqt4/assignment_loss‚1.2895909976365605 /loss/FBt1/assignment_loss‚0.529690071552371 /loss/FRt1/detection_loss‚2.7813482121987776 .loss/FRt3/detection_loss‚2.828950728240766 /loss/SRqqt3/detection_loss‚2.11375892162323 /loss/FBt2/detection_loss‚0.9934644727189033 /loss/FBt4/detection_loss‚3.3055553793907166 0loss/FRt4/assignment_loss‚2.1014610155304867 1loss/SRqqt3/assignment_loss‚2.087014602763312 1loss/SRqqt4/detection_loss‚1.0790759134825652 _step‚102fÞ&œž²‚(:“ ïÒëÔ£ÝòÍ gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.467' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.293400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒAaó‚(:Ý ïÒëÔ¹¡ñÏ( disk./.usagePercent‚79.1212190144865% disk./.usageGB‚1178.9633407592773 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4784.83203125 disk.dm-1.out‚ 66.45703125 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 4784.83203125 disk.sda3.out‚ 66.45703125 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 39592.1328125$ memory_percent‚91.92237854003906 disk.dm-1.in‚0 disk.drbd1260.out‚0.265625 disk.sda.out‚ 66.45703125 disk.sda1.in‚0 disk.dm-0.in‚ 4784.83203125 disk.sd“ˆÚbLa2.out‚0 disk.sda3.in‚0% proc.memory.availableMB‚ 1323.4375# proc.memory.rssMB‚ 3057.23828125) proc.memory.percent‚18.65990161895752 cpu‚25.441006481129108 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 39592.1328125" disk.nvme0n1.out‚ 39592.1328125 network.sent‚ 2.078573e+06 network.recv‚681257@Ђ“´‚(:ˆ þÒëÔ¸€“Ë gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.379' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8758 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0({V´Wµ‚(:Ì þÒëÔÔþŠÏ) proc.memory.percent‚18.65990161895752 cpu‚25.43866325061691 network.sent‚ 2.188226e+06) disk./.usagePercent‚79.15491114203773% disk.nvme0c0n1.out‚39764.83984375 disk.nvme0n1.in‚4858.5 disk.sda1.out‚0 disk.dm-1.out‚ 67.1640625 disk.sda2.in‚0$ memory_percent‚91.92254543304443# disk.nvme0n1.out‚39764.83984375 disk.sda.in‚0 disk.sda.out‚ 67.1640625 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚687149 disk.nvme0c0n1.in‚4858.5 disk.sda3.in‚0 disk.sda3.out‚ 67.1640625 disk.dm-0.in‚4858.5 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.265625% disk./.usageGB‚1179.4653778076172 disk.dm-0.out‚39764.83984375) proc.memory.availableMB‚ 1323.41015625# proc.memory.rssMB‚ 3057.23828125ž“s—‘¶‚(:† ÓëÔžáÁÊ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.28' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Ù¤g·‚(:Ü ÓëÔçàÀË network.sent‚ 2.191412e+06) disk./.usagePercent‚79.15319947566381 disk.drbd1260.out‚ 0.26953125( disk.nvme0c0n1.out‚40014.53271484375 disk.sda.in‚0 disk.dm-0.in‚ 4913.71875 disk.dm-1.in‚0$ memory_percent‚91.92261695861816 network.recv‚690735 disk.nvme0n1.in‚ 4913.71875 disk.sda3.in‚0# disk.dm-0.out‚40014.53271484375( proc.memory.availableMB‚ 1323.3984375) proc.memory.percent‚18.65990161895752 cpu‚25.43654495190884% disk./.usageGB‚1179.4398727416992 disk.sda2.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚40014.53271484375 disk.sda.out‚67.59375 disk.sda1.in‚0# proc.memory.rssMB‚ 3057.23828125 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 4913.71875 disk.sda3.out‚67.59375 disk.sda1.out‚0 disk.dm-1.out‚67.59375 proc.cpu.threads‚17ºö“‘¸‚(:† œÓëÔÕÙáË gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.83' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.966 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰­`žo¹‚(:ä œÓëÔÑ‘´Ð disk.nvme0n1.in‚ 4913.71875 disk.sda3.in‚0 disk.sda3.out‚ 68.56640625 disk.sda.out‚ 68.56640625 disk.sda1.out‚0 disk.dm-1.out‚ 68.56640625$ memory_percent‚91.92419052124023& disk.nvme0n1.out‚40508.43505859375 network.recv‚694953) disk./.usagePercent‚79.15477443401446 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 4913.71875 disk.sda1.in‚0 disk.dm-0.in‚ 4913.71875 disk.dm-1.in‚0 network.sent‚ 2.197383e+06# disk.dm-0.out‚40508.43505859375# proc.memory.rssMB‚ 3057.23828125) proc.memory.percent‚18.65990161895752 cpu‚25.43460308741348 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.26953125( disk.nvme0c0n1.out‚40508.43505859375 disk.sda.in‚0' proc.memory.availableMB‚ 1323.140625 proc.cpu.threads‚17% disk./.usageGB‚1179.4633407592773@ºc“º‚(:ˆ «ÓëÔ«É¸Í gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.759' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èàÄT»‚(:É «ÓëÔÑöÝÏ disk.drbd1260.in‚0 disk.nvme0n1.in‚ 4913.71875 disk.nvme0n1.out‚ 40907.8125 disk.sda3.out‚ 69.484375 disk.sda1.in‚0 disk.dm-0.in‚ 4913.71875 disk.sda2.in‚0' proc.memory.availableMB‚ 1323.390625 network.sent‚ 2.203553e+06 network.recv‚699928% disk./.usageGB‚1179.4633407592773 disk.drbd1260.out‚ 0.2734375! disk.nvme0c0n1.out‚ 40907.8125 disk.sda.out‚ 69.484375 disk.sda1.out‚0 disk.dm-0.out‚ 40907.8125) disk./.usagePercent‚79.15477443401446 disk.nvme0c0n1.in‚ 4913.71875 disk.dm-1.in‚0 disk.dm-1.out‚ 69.484375 disk.sda2.out‚0$ memory_percent‚91.92266464233398# proc.memory.rssMB‚ 3057.23828125) proc.memory.percent‚18.65990161895752 disk.sda3.in‚0 disk.sda.in‚0 cpu‚25.432114612151793 proc.cpu.threads‚17˜‹õ²2¼- _wandbruntime‚2057  _runtime‚2057¾·äǽÁ % _timestamp‚1.7885372620850937e+09 _runtime‚2057.396118759 _step‚103 & lr-AdamW/pg1‚0.0016852081904887286 & lr-AdamW/pg2‚0.0016852081904887286 trainer/global_step‚1299®¼ë/˾Š& lr-AdamW/pg1‚0.0016852081904887286 & lr-AdamW/pg2‚0.0016852081904887286 trainer/global_step‚1299 % _timestamp‚1.7885372620850937e+09 _runtime‚2057.396118759 _step‚103gìülø2¿- _wandbruntime‚2057  _runtime‚20578’Þùë'ÀåO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32940509 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚2057.397996218 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27193782 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32940509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111035871262720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109547105515008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1560428647936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1488783750656 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111035871262720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122419457779712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1489693721088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1484123889681 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 $ _timestamp‚1.788537262086713e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16293956 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60134291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27193780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109547087512064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107405693078657 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1489693721088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚1299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24269259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122419714697216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107404768064301 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 _step‚104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111036799236096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1483215914685 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7974677 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1560195670528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109547105515008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚329141679Ðï'ÁéO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27193782 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32914167 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7975303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120859286049280 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24268631 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27193780 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109547087512064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111035871262720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109547105515008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1560428647936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1488783750656 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111035871262720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122419457779712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1489693721088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1484123889681 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 $ _timestamp‚1.788537262086713e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16293956 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60134291 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27193780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109547087512064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107405693078657 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1489693721088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 trainer/global_step‚1299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24269259 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122419714697216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107404768064301 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 _step‚104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111036799236096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1483215914685 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7974677 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1560195670528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109547105515008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32914167 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32940509 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 _runtime‚2057.397996218 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27193782 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32940509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1489905210880 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7981026 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚2058.793802876 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 % _timestamp‚1.7885372634787211e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32965366 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60179308 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32938998 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106000569744160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109628724600832 _step‚105 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109628724600832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24287047 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109628706597888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107484902932123 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27213940 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1561358345728 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24286424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16306021 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728t#ðì„385 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122510846159360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106000586783520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111118611808768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122511100966400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16306019 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111119539795456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27213942 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27213940 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1489905210880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1490815194624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109628706597888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1561589212672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120949511753728žh«ú&ÄôM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109628706597888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1561589212672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120949511753728 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1489905210880 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7981026 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚2058.793802876 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 % _timestamp‚1.7885372634787211e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32965366 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60179308 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32938998 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106000569744160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109628724600832 _step‚105 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109628724600832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24287047 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109628706597888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107484902932123 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27213940 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1561358345728 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24286424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16306021 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122510846159360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106000586783520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111118611808768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122511100966400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16306019 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111119539795456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27213942 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27213940 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1489905210880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1490815194624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚1299iÔ½· 2Å- _wandbruntime‚2058  _runtime‚2058òäéXÆÒ /loss/FRt1/assignment_loss‚5.500355396005842 1loss/FBt3/assignment_loss‚0.23975071416318627 /loss/FRt2/detection_loss‚2.0619669148455495 /loss/FRt3/detection_loss‚2.8628388392297843 1loss/SRqqt3/detection_loss‚2.1056441905650685 /loss/FBt3/detection_loss‚2.6626475084395635 0loss/FBt1/assignment_loss‚0.5746034698094462 $loss/total_loss‚51.4136409223503 epoch‚1 _step‚106 .loss/FRt1/detection_loss‚2.672139277888669 .loss/FBt1/detection_loss‚2.477423332631588 % _timestamp‚1.7885372634855294e+09 0loss/FRt2/assignment_loss‚2.5038906128510185 1loss/FBt2/assignment_loss‚0.16077005740612305 0loss/SRqqt1/detection_loss‚2.575369954109192 /loss/FBt2/detection_loss‚1.0022537656926682 .loss/FBt4/detection_loss‚3.168217968940735 _runtime‚2058.796550125 /loss/FRt3/assignment_loss‚4.561922487459685 1loss/SRqqt4/assignment_loss‚1.401619174047909 .loss/FRt4/detection_loss‚1.611908852033174 1loss/SRqqt4/detection_loss‚1.2264191678403753 trainer/global_step‚1299 0loss/FRt4/assignment_loss‚2.8984423249195785 2loss/SRqqt1/assignment_loss‚1.8591068675120672 1loss/SRqqt2/assignment_loss‚2.143883435165181 2loss/SRqqt3/assignment_loss‚1.8276812232872273 1loss/FBt4/assignment_loss‚0.33830374997269247 1loss/SRqqt2/detection_loss‚2.9764816375339733Ã9¦\ÇÖ 0loss/FBt1/assignment_loss‚0.5746034698094462 $loss/total_loss‚51.4136409223503 epoch‚1 _step‚106 .loss/FRt1/detection_loss‚2.672139277888669 .loss/FBt1/detection_loss‚2.477423332631588 % _timestamp‚1.7885372634855294e+09 0loss/FRt2/assignment_loss‚2.5038906128510185 1loss/FBt2/assignment_loss‚0.16077005740612305 0loss/SRqqt1/detection_loss‚2.575369954109192 /loss/FBt2/detection_loss‚1.0022537656926682 .loss/FBt4/detection_loss‚3.168217968940735 _runtime‚2058.796550125 /loss/FRt3/assignment_loss‚4.561922487459685 1loss/SRqqt4/assignment_loss‚1.401619174047909 .loss/FRt4/detection_loss‚1.611908852033174 1loss/SRqqt4/detection_loss‚1.2264191678403753 trainer/global_step‚1299 0loss/FRt4/assignment_loss‚2.8984423249195785 2loss/SRqqt1/assignment_loss‚1.8591068675120672 1loss/SRqqt2/assignment_loss‚2.143883435165181 2loss/SRqqt3/assignment_loss‚1.8276812232872273 1loss/FBt4/assignment_loss‚0.33830374997269247 1loss/SRqqt2/detection_loss‚2.9764816375339733 /loss/FRt1/assignment_loss‚5.500355396005842 1loss/FBt3/assignment_loss‚0.23975071416318627 /loss/FRt2/detection_loss‚2.0619669148455495 /loss/FRt3/detection_loss‚2.8628388392297843 1loss/SRqqt3/detection_loss‚2.1056441905650685 /loss/FBt3/detection_loss‚2.6626475084395635jüýUM‘È‚(:† ºÓëÔ‘žÐÊ gpu.0.gpu‚25.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.39' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.278 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î×áhÉ‚(:Ý ºÓëÔ’±øÌ disk.sda1.out‚0$ memory_percent‚91.92516803741455) proc.memory.percent‚18.65990161895752 cpu‚25.430067989842648 network.sent‚ 2.208506e+06 disk.sda.out‚ 69.921875 disk.dm-1.in‚0 disk.dm-1.out‚ 69.921875 proc.cpu.threads‚17 disk.sda3.out‚ 69.921875) disk./.usagePercent‚79.12879427742467% disk./.usageGB‚1179.0762176513672 disk.sda2.out‚0' disk.nvme0c0n1.out‚41119.0029296875 disk.nvme0n1.in‚ 4913.71875% disk.nvme0n1.out‚41119.0029296875) proc.memory.availableMB‚ 1322.98046875 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2734375 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 4913.71875" disk.dm-0.out‚41119.0029296875# proc.memory.rssMB‚ 3057.23828125 disk.sda3.in‚0 network.recv‚703274 disk.nvme0c0n1.in‚ 4913.71875M÷ž•’Ê‚(:‡ ÉÓëÔÀ¥„Ë gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.525' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.105 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÌE‘bË‚(:× ÉÓëÔðäÎ cpu‚25.427819149712622# disk./.usageGB‚1179.07612991333 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0n1.out‚41333.79296875 disk.sda1.in‚0# proc.memory.rssMB‚ 3057.23828125) proc.memory.percent‚18.65990161895752 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.27734375 disk.nvme0n1.in‚ 5055.71875 disk.sda3.out‚ 70.7109375 disk.dm-0.out‚41333.79296875 disk.dm-1.in‚0( proc.memory.availableMB‚ 1323.1796875 network.sent‚ 2.31614e+06 network.recv‚708542) disk./.usagePercent‚79.12878838925138 disk.nvme0c0n1.in‚ 5055.71875 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 70.7109375 disk.dm-1.out‚ 70.7109375 disk.drbd1260.in‚0% disk.nvme0c0n1.out‚41333.79296875 disk.sda1.out‚0 disk.dm-0.in‚ 5055.71875$ memory_percent‚91.92395210266113¡Ç/Ì‚(:„ ØÓëÔƒ€êÌ gpu.0.gpu‚19.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.9' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã®2hÍ‚(:Ý ØÓëÔÇÓÍÍ disk.drbd1260.out‚ 0.27734375 disk.nvme0c0n1.in‚ 5055.71875 disk.sda3.out‚ 71.43359375 disk.sda.out‚ 71.43359375) proc.memory.percent‚18.65990161895752 cpu‚25.425792148895923 network.sent‚ 2.322117e+06 network.recv‚712994) disk./.usagePercent‚79.12878454913837# disk.nvme0n1.out‚41631.44140625 disk.sda1.out‚0 disk.dm-0.out‚41631.44140625 disk.dm-1.in‚0 disk.dm-1.out‚ 71.43359375$ disk./.usageGB‚1179.076072692871 disk.sda2.in‚0% disk.nvme0c0n1.out‚41631.44140625 disk.sda.in‚0$ memory_percent‚91.92397594451904) proc.memory.availableMB‚ 1323.17578125# proc.memory.rssMB‚ 3057.23828125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 5055.71875 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 5055.71875óƒ‡ôžÎ‚(:“ çÓëÔ³ž÷Ì gpu.0.gpu‚37.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.264' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.652800000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`\à9lÏ‚(:á çÓëÔ¨þªÏ# proc.memory.rssMB‚ 3057.23828125 disk.sda2.in‚0 disk.dm-0.in‚ 5055.71875) proc.memory.percent‚18.65990161895752 proc.cpu.threads‚17 network.recv‚717730) disk./.usagePercent‚79.12721496694594 disk.drbd1260.out‚0.28125 disk.nvme0n1.in‚ 5055.71875 disk.sda3.in‚0 disk.sda3.out‚ 72.2734375 disk.dm-1.in‚0 cpu‚25.423721828948104 network.sent‚ 2.328284e+06% disk./.usageGB‚1179.0526847839355 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚ 5055.71875& disk.nvme0n1.out‚42039.02099609375 disk.sda.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚42039.02099609375 disk.sda.out‚ 72.2734375 disk.sda1.in‚0# disk.dm-0.out‚42039.02099609375 disk.dm-1.out‚ 72.2734375$ memory_percent‚91.92450046539307) proc.memory.availableMB‚ 1323.08984375Mœ²2Ð- _wandbruntime‚2125  _runtime‚2125æD ÄѾ % lr-AdamW/pg1‚0.001684804318924365 % lr-AdamW/pg2‚0.001684804318924365 trainer/global_step‚1349 $ _timestamp‚1.788537330039943e+09 _runtime‚2125.351478388 _step‚107ô™n³ÈÒ _step‚107 % lr-AdamW/pg1‚0.001684804318924365 % lr-AdamW/pg2‚0.001684804318924365 trainer/global_step‚1349 $ _timestamp‚1.788537330039943e+09 _runtime‚2125.351478388k9ë–ø2Ó- _wandbruntime‚2125  _runtime‚2125s;ì'ÔæO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1545781146112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125370547331584 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚2125.355217016 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115172912978944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25156674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259014656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115172912978944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113628059806208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1544871175680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1539093169793 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62360849 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28201780 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113628041803264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25156068 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115173840952320 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34185411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16897204 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28201780 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34185411 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34159069 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111411525046609 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16897206 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115172912978944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25156674 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259014656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115172912978944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235074560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113628059806208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1544871175680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1539093169793 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62360849 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28201780 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113628041803264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25156068 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115173840952320 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34185411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16897204 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28201780 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760lÕn2Ö- _wandbruntime‚2126  _runtime‚2126¦ß<ô&×îM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125460773036032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111492584928995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34183900 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1546902619648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127080610296832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2126.827061239 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115255653524992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115256581511680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1541118418275 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127080361781248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚109 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1545992635904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62432210 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16909271 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1546902619648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28221942 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28221942 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113709678892032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115256581511680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28221940 trainer/global_step‚1349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34183900 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25174442 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62405840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1619588745216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111491659914431 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109951466510720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62432210 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25173832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1540210443071 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34210268 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113709678892032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8265171 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8264563 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1619813320704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109951449471360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28221940 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113709660889088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1545992635904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 # _timestamp‚1.78853733151495e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113709660889088 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25173832 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1540210443071 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34210268 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113709678892032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8265171 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8264563 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1619813320704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109951449471360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28221940 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113709660889088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1545992635904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 # _timestamp‚1.78853733151495e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113709660889088 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34183900 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1546902619648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127080610296832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2126.827061239 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115255653524992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115256581511680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1541118418275 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127080361781248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚109 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1545992635904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62432210 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16909271 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1546902619648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28221942 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28221942 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113709678892032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115256581511680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28221940 trainer/global_step‚1349 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34183900 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25174442 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62405840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1619588745216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111491659914431 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109951466510720 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62432210 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8545646 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17500454 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1600958600704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113823311604016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1676965100032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1594970424901 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117709014097408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117708996094464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64640095 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚2192.268897126 _step‚112 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119309954695168 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119309954695168 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8545646 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17500454 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1600958600704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113823311604016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1676965100032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1594970424901 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117709014097408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚117708996094464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64640095 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 _runtime‚2192.268897126 _step‚112 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64685112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1602990044672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119392695241216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117790615180288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35455170 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117790615180288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602080060928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1602990044672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64685112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35455170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17512519 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113902329198560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚113 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17512521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 PDeviceStatsMonitor.on_train_batch_end/inactive_sp°‚^lit.all.freed‚26063266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131650212366848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29229942 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596995673383 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1602080060928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1677909787136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115498416896739 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29229942 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1596087698179 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚2193.487212201 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117790633183232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8551345 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129972082198528 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244321280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26063866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119393623227904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8550747 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131649968045568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35455170 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117790615180288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602080060928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1602990044672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64685112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35455170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17512519 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113902329198560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚113 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17512521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26063266 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚131650212366848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29229942 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596995673383 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1602080060928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1677909787136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115498416896739 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29229942 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1596087698179 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 _runtime‚2193.487212201 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117790633183232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8551345 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129972082198528 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244321280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26063866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119393623227904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8550747 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131649968045568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64685112qÐ8k«2ï- _wandbruntime‚2193  _runtime‚2193»Æ®OYðÓ 2loss/SRqqt3/assignment_loss‚2.4126246100471866 0loss/FRt2/assignment_loss‚2.7305139093785673 /loss/FRt4/assignment_loss‚2.230310155586763 0loss/FBt1/assignment_loss‚0.2987387636665062 /loss/FBt1/detection_loss‚2.7205554097890854 /loss/FBt2/detection_loss‚0.9204108527329351 0loss/FBt4/assignment_loss‚0.2604818299296312 /loss/FRt1/detection_loss‚2.9675885010510683 /loss/FBt3/detection_loss‚2.7451621572176617 $ _timestamp‚1.788537398179018e+09 _step‚114 /loss/FRt1/assignment_loss‚4.772682126611471 2loss/SRqqt4/assignment_loss‚1.4279586106422357 1loss/SRqqt2/detection_loss‚2.4215718979636827 %loss/total_loss‚51.42675776909946 trainer/global_step‚1399 _runtime‚2193.489946445 2loss/SRqqt2/assignment_loss‚2.4521377831697464 1loss/FBt2/assignment_loss‚0.19297448758977598 1loss/SRqqt4/detection_loss‚0.9371118713752367 /loss/FRt3/assignment_loss‚4.607845419929141 /loss/FRt3/detection_loss‚2.7317845537548973 /loss/FRt4/detection_loss‚1.7392377368682488 epoch‚1 /loss/FRt2/detection_loss‚2.4505553334145933 1loss/SRqqt1/assignment_loss‚1.876306061943372 0loss/FBt3/assignment_loss‚0.1569371263836956 1loss/SRqqt1/detection_loss‚2.6511184692382814 0loss/SRqqt3/detection_loss‚2.629947973835853 .loss/FBt4/detection_loss‚3.092202126979828zw¬]ñ× /loss/FRt4/assignment_loss‚2.230310155586763 0loss/FBt1/assignment_loss‚0.2987387636665062 /loss/FBt1/detection_loss‚2.7205554097890854 /loss/FBt2/detection_loss‚0.9204108527329351 0loss/FBt4/assignment_loss‚0.2604818299296312 /loss/FRt1/detection_loss‚2.9675885010510683 /loss/FBt3/detection_loss‚2.7451621572176617 $ _timestamp‚1.788537398179018e+09 _step‚114 /loss/FRt1/assignment_loss‚4.772682126611471 2loss/SRqqt4/assignment_loss‚1.4279586106422357 1loss/SRqqt2/detection_loss‚2.4215718979636827 %loss/total_loss‚51.42675776909946 trainer/global_step‚1399 _runtime‚2193.489946445 2loss/SRqqt2/assignment_loss‚2.4521377831697464 1loss/FBt2/assignment_loss‚0.19297448758977598 1loss/SRqqt4/detection_loss‚0.9371118713752367 /loss/FRt3/assignment_loss‚4.607845419929141 /loss/FRt3/detection_loss‚2.7317845537548973 /loss/FRt4/detection_loss‚1.7392377368682488 epoch‚1 /loss/FRt2/detection_loss‚2.4505553334145933 1loss/SRqqt1/assignment_loss‚1.876306061943372 0loss/FBt3/assignment_loss‚0.1569371263836956 1loss/SRqqt1/detection_loss‚2.6511184692382814 0loss/SRqqt3/detection_loss‚2.629947973835853 .loss/FBt4/detection_loss‚3.092202126979828 2loss/SRqqt3/assignment_loss‚2.4126246100471866 0loss/FRt2/assignment_loss‚2.7305139093785673r§gS‘ò‚(:† ÁÔëÔ¦‹Î gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥}WÉkó‚(:à ÁÔëÔÎܪÐ network.recv‚742528# disk.dm-0.out‚43812.59228515625 disk.dm-1.out‚ 76.1171875 disk.drbd1260.in‚0) proc.memory.percent‚18.65992546081543 disk.dm-0.in‚ 5376.890625 network.sent‚ 2.462626e+06 disk.sda3.in‚0 disk.sda3.out‚ 76.1171875 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.296875! disk.nvme0c0n1.in‚ 5376.890625) disk./.usagePercent‚79.12883498262262 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚43812.59228515625$ memory_percent‚91.92543029785156% proc.memory.availableMB‚ 1322.9375" proc.memory.rssMB‚ 3057.2421875 disk.sda1.out‚0% disk./.usageGB‚1179.0768241882324 disk.nvme0n1.in‚ 5376.890625& disk.nvme0n1.out‚43812.59228515625 disk.sda.out‚ 76.1171875 cpu‚25.411878759116274 proc.cpu.threads‚17ß‚c“ô‚(:ˆ ÐÔëÔ¥ê²Í gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚116.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘0·¼iõ‚(:Þ ÐÔëÔ—­²Ï) disk./.usagePercent‚79.13037896406162% disk./.usageGB‚1179.0998306274414 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.95982646942139 cpu‚25.410058630268693 network.sent‚ 2.572984e+06& disk.nvme0n1.out‚44220.07666015625 disk.sda3.out‚ 76.671875 disk.sda2.out‚0" proc.memory.rssMB‚ 3057.2421875! disk.nvme0c0n1.in‚ 5376.890625( disk.nvme0c0n1.out‚44220.07666015625 disk.sda1.out‚0 disk.dm-0.in‚ 5376.890625# disk.dm-0.out‚44220.07666015625 disk.dm-1.in‚0 disk.dm-1.out‚ 76.671875 disk.drbd1260.out‚0.296875 network.recv‚748826 disk.nvme0n1.in‚ 5376.890625 disk.sda3.in‚0 disk.sda.out‚ 76.671875 disk.sda1.in‚0& proc.memory.availableMB‚ 6.58203125) proc.memory.percent‚18.65992546081543 proc.cpu.threads‚17o@\s]ö‚(:Ò ßÔëÔϳØÌ disk.dm-1.in‚0 disk.dm-1.out‚77.25 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚44283.52197265625 disk.sda3.in‚0 disk.sda3.out‚77.25 disk.drbd1260.out‚ 0.30078125$ memory_percent‚99.95529651641846" proc.memory.rssMB‚ 3057.2421875 cpu‚25.40819380090315 network.recv‚754095) disk./.usagePercent‚79.13038152413696$ disk./.usageGB‚1179.099868774414 disk.sda1.out‚0# disk.dm-0.out‚44283.52197265625 disk.sda2.in‚0& disk.nvme0n1.out‚44283.52197265625& proc.memory.availableMB‚ 7.32421875 network.sent‚ 2.579566e+06 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 5376.890625 disk.nvme0n1.in‚ 5376.890625) proc.memory.percent‚18.65992546081543 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚77.25 disk.sda1.in‚0 disk.dm-0.in‚ 5376.890625^™ºÄž÷‚(:“ ßÔëÔŸ¨±Ð gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.497' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.299400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.UÞ“ø‚(:ˆ îÔëÔ“«©Í gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ”Õmù‚(:â îÔëÔ½”éÏ) proc.memory.percent‚18.65992546081543% disk./.usageGB‚1179.0764198303223 disk.sda.out‚ 77.6640625 disk.sda1.out‚0# disk.dm-0.out‚44681.36181640625 disk.sda2.in‚0% proc.memory.availableMB‚ 7.0703125" proc.memory.rssMB‚ 3057.2421875! disk.nvme0c0n1.in‚ 5376.890625( disk.nvme0c0n1.out‚44681.36181640625 disk.dm-0.in‚ 5376.890625 cpu‚25.406437279395966 proc.cpu.threads‚17) disk./.usagePercent‚79.12880784582399 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 5376.890625 disk.sda3.out‚ 77.6640625 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 77.6640625$ memory_percent‚99.95684623718262 network.sent‚ 2.584553e+06 network.recv‚757641 disk.drbd1260.out‚ 0.30078125& disk.nvme0n1.out‚44681.36181640625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0n_é2ú- _wandbruntime‚2258  _runtime‚2258c=ŠÅû¿ % lr-AdamW/pg2‚0.001683470245675064 trainer/global_step‚1449 % _timestamp‚1.7885374633285384e+09 _runtime‚2258.639518283 _step‚115 % lr-AdamW/pg1‚0.001683470245675064ÍØÉüà _step‚115 % lr-AdamW/pg1‚0.001683470245675064 % lr-AdamW/pg2‚0.001683470245675064 trainer/global_step‚1449 % _timestamp‚1.7885374633285384e+09 _runtime‚2258.639518283sâ¯5ó2ý- _wandbruntime‚2258  _runtime‚2258â|O%]%þæO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117774191331216 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30217780 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1735109825024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30217780 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1735342802432 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36648873 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66892997 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30217782 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136128484518912 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121789968388608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66866653 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 trainer/global_step‚1449 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136128227601408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1657046025728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36675215 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26936968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8833262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121789968388608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119425039011225 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123446996411392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚ŠY¨0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚116 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123447924384768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0"ÌÜOð'ÿêO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30217780 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1735109825024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30217780 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1735342802432 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36648873 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66892997 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30217782 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136128484518912 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121789968388608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66866653 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 trainer/global_step‚1449 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136128227601408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1657046025728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36675215 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26936968 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8833262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121789968388608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119425039011225 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123446996411392 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚116 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚123447924384768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117774191331216 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004t¥uß2€- _wandbruntime‚2259  _runtime‚2259=¸™?ö&ðM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36700072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66938014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚2259.643851467 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121871587474432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123530664944128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30237942 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123530664944128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1659077469696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66938014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18115769 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134483343480832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚117 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26954127 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1659077469696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26954728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1652872928491 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66911644 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121871569471488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36673704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117853208925760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117853225965120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121871587474432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1658167485952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8838358 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885374643323598e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123529736957440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136219619423744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1651964953287 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30237940 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136219872133632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119505173879047 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1736504712704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36700072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18115771 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119506098893611 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36673704 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26954127 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1659077469696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26954728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1652872928491 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66911644 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121871569471488 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36673704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117853208925760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117853225965120 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121871587474432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1658167485952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8838358 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885374643323598e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123529736957440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136219619423744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1651964953287 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30237940 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136219872133632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119505173879047 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1736504712704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36700072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18115771 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119506098893611 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36673704 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66938014 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚2259.643851467 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121871587474432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123530664944128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30237942 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123530664944128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1659077469696 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66938014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18115769 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134483343480832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚117 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1707632910113 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125870922679808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31225782 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚1499 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1793621456896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37893775 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27828360 $ _timestamp‚1.788537523573002e+09 _runtime‚2318.884248598 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127584038127616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18706956 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123432721007889 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31225780 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127584966100992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125870904676864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18706954 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138904402998784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1714043421184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140698270887424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1793843948544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31225782 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121725071058416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1706724935117 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1714043421184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69145899 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69119555 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31225780 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27828964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37920117 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222491648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1713133450752 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9122008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127584038127616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37920117 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9121406 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123431795993533 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125870904676864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1713133450752 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1707632910113 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125870922679808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31225782 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 trainer/global_step‚1499 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1793621456896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37893775 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27828360 $ _timestamp‚1.788537523573002e+09 _runtime‚2318.884248598 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127584038127616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18706956 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123432721007889 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31225780 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127584966100992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125870904676864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18706954 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138904402998784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1714043421184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140698270887424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1793843948544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31225782 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121725071058416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1706724935117 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1714043421184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69145899 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69119555 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31225780 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27828964 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37920117 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222491648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1713133450752 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9122008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127584038127616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37920117 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9121406 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123431795993533 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125870904676864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1713133450752xW„¦2”- _wandbruntime‚2320  _runtime‚2320K”ިÕðM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125952541765632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69164546 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1715164894720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27846163 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69190916 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127666778673664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138994628703232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123512855875919 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127667706660352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69190916 % _timestamp‚1.7885375248959184e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140789667983360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123511930861355 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140789413176320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1708750183599 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125952541765632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121804088652960 _step‚121 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31245942 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9127752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1707842208395 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37918606 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138994652643328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1795015340032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31245942 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125952523762688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69164546 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18719021â}mú&–ôM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37944974 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1714254910976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37944974 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1794784473088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚127667706660352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9127144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31245942 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9127752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1707842208395 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37918606 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138994652643328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1795015340032 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31245942 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125952523762688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69164546 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18719021 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125952541765632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69164546 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1715164894720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27846163 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69190916 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127666778673664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138994628703232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123512855875919 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127667706660352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69190916 % _timestamp‚1.7885375248959184e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140789667983360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123511930861355 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140789413176320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1708750183599 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125952541765632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121804088652960 _step‚121 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129951876971008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129951858968064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131722007817216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39138677 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852075797504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚124 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131721079843840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1769220875776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145267764018688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143415688221184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125675950785616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131721079843840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39165019 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1770130846208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1769220875776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129951876971008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19310206 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145268016741888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1762602190225 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004pÿ=î'§èO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1770130846208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1769220875776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129951876971008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19310206 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145268016741888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1762602190225 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131722007817216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1770130846208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71398801 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28721934 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39165019 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127438552975841 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129951858968064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71372457 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125675967824976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1763510165221 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39138677 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129951876971008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129951858968064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131722007817216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39138677 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1852075797504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚124 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131721079843840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1769220875776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145267764018688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143415688221184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125675950785616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131721079843840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39165019 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0|0„T2¨- _wandbruntime‚2379  _runtime‚2379…Û ô&©îM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71443818 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32253940 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1853257998336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127519612858227 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1770342336000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9417390 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853478379520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9417995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71443818 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127518687843663 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32253942 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39163508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125754968380160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39189876 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1771252319744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71417448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145359416245248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130033496056832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39189876 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130033478053888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1771252319744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28740266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28739659 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131804748376576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19322271 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125754985419520 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764627438707 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚125 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71417448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32253940 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143505913925632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39163508 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19322269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1770342336000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145359171923968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130033478053888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ÜJ4pø&ªòM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32253942 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39163508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125754968380160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39189876 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1771252319744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71417448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145359416245248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130033496056832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39189876 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130033478053888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1771252319744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28740266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28739659 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131804748376576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19322271 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125754985419520 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764627438707 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚125 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71417448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32253940 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143505913925632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39163508 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19322269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1770342336000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145359171923968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130033478053888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71443818 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32253940 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1853257998336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127519612858227 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1770342336000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9417390 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1853478379520 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9417995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71443818 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127518687843663 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784}Dl2«- _wandbruntime‚2379  _runtime‚2379KÒ´!Y¬Ó /loss/FRt1/assignment_loss‚4.633226535537026 0loss/FRt3/assignment_loss‚4.2077975483501655 2loss/SRqqt1/assignment_loss‚2.1330035317827156 1loss/SRqqt4/assignment_loss‚1.497872144422637 /loss/FRt2/detection_loss‚2.6951594459824264 1loss/SRqqt2/detection_loss‚3.6574224737974315 .loss/FBt3/detection_loss‚2.511446329454581 2loss/SRqqt2/assignment_loss‚2.1189453418438253 1loss/FBt1/assignment_loss‚0.29532683131406684 1loss/FBt2/assignment_loss‚0.21222223117189712 %loss/total_loss‚52.21879261104202 trainer/global_step‚1549 _runtime‚2379.762016381 _step‚126 0loss/FBt4/assignment_loss‚0.2545071420025933 1loss/SRqqt4/detection_loss‚1.1884699219551225 epoch‚1 2loss/SRqqt3/assignment_loss‚1.9399807502826054 .loss/FRt1/detection_loss‚2.840887492353266 1loss/SRqqt3/detection_loss‚2.3038539304619743 /loss/FBt4/detection_loss‚3.0743017196655273 /loss/FRt4/assignment_loss‚2.469406393292832 0loss/FBt3/assignment_loss‚0.3658654054161161 /loss/FRt4/detection_loss‚1.7089148806408048 /loss/FBt2/detection_loss‚0.8387024337564831 % _timestamp‚1.7885375844510994e+09 0loss/FRt2/assignment_loss‚3.3945385832339525 .loss/FRt3/detection_loss‚2.812467476900886 0loss/SRqqt1/detection_loss‚2.727174096637302 .loss/FBt1/detection_loss‚2.337299970785777Ûb©Ÿ­× 0loss/FRt3/assignment_loss‚4.2077975483501655 2loss/SRqqt1/assignment_loss‚2.1330035317827156 1loss/SRqqt4/assignment_loss‚1.497872144422637 /loss/FRt2/detection_loss‚2.6951594459824264 1loss/SRqqt2/detection_loss‚3.6574224737974315 .loss/FBt3/detection_loss‚2.511446329454581 2loss/SRqqt2/assignment_loss‚2.1189453418438253 1loss/FBt1/assignment_loss‚0.29532683131406684 1loss/FBt2/assignment_loss‚0.21222223117189712 %loss/total_loss‚52.21879261104202 trainer/global_step‚1549 _runtime‚2379.762016381 _step‚126 0loss/FBt4/assignment_loss‚0.2545071420025933 1loss/SRqqt4/detection_loss‚1.188469921955g‹¾1225 epoch‚1 2loss/SRqqt3/assignment_loss‚1.9399807502826054 .loss/FRt1/detection_loss‚2.840887492353266 1loss/SRqqt3/detection_loss‚2.3038539304619743 /loss/FBt4/detection_loss‚3.0743017196655273 /loss/FRt4/assignment_loss‚2.469406393292832 0loss/FBt3/assignment_loss‚0.3658654054161161 /loss/FRt4/detection_loss‚1.7089148806408048 /loss/FBt2/detection_loss‚0.8387024337564831 % _timestamp‚1.7885375844510994e+09 0loss/FRt2/assignment_loss‚3.3945385832339525 .loss/FRt3/detection_loss‚2.812467476900886 0loss/SRqqt1/detection_loss‚2.727174096637302 .loss/FBt1/detection_loss‚2.337299970785777 /loss/FRt1/assignment_loss‚4.633226535537026~`[k¾“®‚(:ˆ õÕëÔ“ðúÍ gpu.0.gpu‚21.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.218' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰óó^d¯‚(:Ù õÕëÔÓ½ÒÏ$ memory_percent‚99.95801448822021) proc.memory.percent‚18.65997314453125 cpu‚25.39067250869023 disk.sda3.in‚0 disk.sda3.out‚ 82.7734375 network.recv‚797312 disk.sda2.in‚0 disk.drbd1260.in‚0# disk.nvme0n1.out‚46823.64453125 disk.sda.out‚ 82.7734375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 5640.37890625% disk.nvme0c0n1.out‚46823.64453125 disk.dm-0.out‚46823.64453125 disk.dm-1.out‚ 82.7734375 proc.memory.rssMB‚3057.25 proc.cpu.threads‚17) disk./.usagePercent‚79.13042197332737$ disk./.usageGB‚1179.100471496582 disk.sda2.out‚0! disk.nvme0n1.in‚ 5640.37890625 disk.sda1.out‚0 disk.dm-0.in‚ 5640.37890625& proc.memory.availableMB‚ 6.87890625 network.sent‚ 2.837545e+06 disk.drbd1260.out‚ 0.32421875 disk.sda.in‚0 disk.dm-1.in‚0pÔϤž°‚(:“ „ÖëÔÎñšË gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚158.397' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.679399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rLúÒi±‚(:Þ „ÖëÔàôîÍ) disk./.usagePercent‚79.13041838922189 disk.dm-0.out‚46977.00390625 disk.dm-1.in‚0 disk.dm-1.out‚ 83.35546875 disk.sda2.in‚0 disk.sda3.out‚ 83.35546875 disk.sda.out‚ 83.35546875 disk.sda1.out‚0 network.recv‚803808% disk./.usageGB‚1179.1004180908203 disk.drbd1260.out‚ 0.32421875# disk.nvme0c0n1.in‚ 5831.98828125% disk.nvme0c0n1.out‚46977.00390625! disk.nvme0n1.in‚ 5831.98828125 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 5831.98828125 disk.sda1.in‚0$ memory_percent‚99.95968341827393& proc.memory.availableMB‚ 6.60546875 proc.memory.rssMB‚3057.25 cpu‚25.389134676238942 proc.cpu.threads‚17 network.sent‚ 2.947999e+06 disk.sda2.out‚0 disk.drbd1260.in‚0# disk.nvme0n1.out‚46977.00390625) proc.memory.percent‚18.65997314453125&]ˆ“²‚(:ˆ “ÖëÔʸ¨Ì gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.218' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šÀàk³‚(:à “ÖëÔ÷—ŠÐ# disk.nvme0c0n1.in‚ 5831.98828125 disk.dm-1.in‚0 disk.dm-1.out‚ 84.15234375$ memory_percent‚99.95813369750977 cpu‚25.387428776769763 network.recv‚808733 disk.sda2.out‚0' disk.nvme0c0n1.out‚47296.3095703125% disk.nvme0n1.out‚47296.3095703125 disk.sda3.in‚0 disk.sda3.out‚ 84.15234375 disk.sda.in‚0 disk.sda1.in‚0) disk./.usagePercent‚79.12884727098427 disk.drbd1260.out‚0.359375! disk.nvme0n1.in‚ 5831.98828125 disk.sda1.out‚0 disk.dm-0.in‚ 5831.98828125$ proc.memory.availableMB‚6.859375 proc.memory.rssMB‚3057.25) proc.memory.percent‚18.65997314453125% disk./.usageGB‚1179.0770072937012 disk.sda.out‚ 84.15234375" disk.dm-0.out‚47296.3095703125 proc.cpu.threads‚17 network.sent‚ 2.952279e+06 disk.sda2.in‚0 disk.drbd1260.in‚0l„cø’´‚(:‡ ¢ÖëÔõùÍ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚177.925' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nléjµ‚(:ß ¢ÖëÔÎÆÏ network.sent‚ 2.957302e+06 network.recv‚812356 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 5831.98828125' disk.nvme0c0n1.out‚47581.8408203125% disk.nvme0n1.out‚47581.8408203125 disk.dm-1.in‚0 proc.memory.rssMB‚3057.25% disk./.usageGB‚1179.1004943847656 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 84.64453125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ proc.memory.availableMB‚6.859375 disk.sda2.in‚0 disk.drbd1260.out‚0.359375! disk.nvme0n1.in‚ 5831.98828125 disk.sda.out‚ 84.64453125 disk.dm-0.in‚ 5831.98828125" disk.dm-0.out‚47581.8408203125$ memory_percent‚99.95813369750977) proc.memory.percent‚18.65997314453125) disk./.usagePercent‚79.13042350937258 disk.dm-1.out‚ 84.64453125 cpu‚25.38590831198231 proc.cpu.threads‚17¦ü2¶- _wandbruntime‚2438  _runtime‚24386ÑÏ-Æ·À $ _timestamp‚1.788537643494058e+09 _runtime‚2438.804938827 _step‚127 & lr-AdamW/pg1‚0.0016801556695271078 & lr-AdamW/pg2‚0.0016801556695271078 trainer/global_step‚1599kOk*Ê¸Ä & lr-AdamW/pg1‚0.0016801556695271078 & lr-AdamW/pg2‚0.0016801556695271078 trainer/global_step‚1599 $ _timestamp‚1.788537643494058e+09 _runtime‚2438.804938827 _step‚127ô G2¹- _wandbruntime‚2438  _runtime‚2438ÿâóÒì'ºæO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149837502606336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40383579 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1819387420329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19913456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33241780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135858121560064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73651703 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9700813 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1818479445333 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33241780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚1599 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 % _timestamp‚1.7885376434950738e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134032813259264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147926997383680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1825308300800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1910529162752 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185ÄܘXñ'»ëO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149837502606336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40383579 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1819387420329 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19913456 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33241780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135858121560064 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73651703 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9700813 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1818479445333 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33241780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 trainer/global_step‚1599 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 % _timestamp‚1.7885376434950738e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134032813259264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147926997383680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1825308300800 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1910529162752 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1€wÝ -2¼- _wandbruntime‚2440  _runtime‚2440Œ6~нðM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19925519 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1819596718611 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135941790092800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129705848107360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40408410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19925521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚2440.032476711 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148017223088128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134114450348032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1599 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826429761024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40434778 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826429761024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚129 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134114432345088 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148017199148032 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885376447207565e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33261942 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135940862106112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9706652 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1911925198336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827339744768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40408410 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocatedœ«ô¥&_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73670350 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827339744768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73696720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134114432345088 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135941790092800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33261940 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29632171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131525444825971 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129705865146720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33261940 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73670350 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827339744768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73696720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134114432345088 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135941790092800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33261940 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29632171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131525444825971 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129705865146720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33261940 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135941790092800 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129705848107360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40408410 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19925521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚2440.032476711 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148017223088128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134114450348032 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚1599 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826429761024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40434778 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826429761024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 _step‚129 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134114432345088 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148017199148032 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885376447207565e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33261942 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135940862106112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9706652 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1911925198336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827339744768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ª”g2¿- _wandbruntime‚2440  _runtime‚2440‡h"XÀÒ /loss/FRt3/assignment_loss‚4.869003895918528 -loss/FBt4/detection_loss‚3.08846802636981 trainer/global_step‚1599 /loss/FRt1/assignment_loss‚4.342697191238403 1loss/SRqqt1/detection_loss‚2.4010330193183003 1loss/SRqqt3/detection_loss‚2.1033736556134324 /loss/FBt2/detection_loss‚0.9852056989477853 % _timestamp‚1.7885376447243648e+09 _step‚130 2loss/SRqqt3/assignment_loss‚1.4605585704458521 1loss/FBt4/assignment_loss‚0.04426601713794298 /loss/FRt4/detection_loss‚1.7176238317787647 1loss/SRqqt4/detection_loss‚1.0785805726604116 %loss/total_loss‚49.28623619632059 1loss/SRqqt1/assignment_loss‚2.297019791953704 .loss/FRt1/detection_loss‚2.847890059649944 .loss/FRt3/detection_loss‚3.141257127126058 /loss/FRt4/assignment_loss‚2.131458195845286 1loss/FBt1/assignment_loss‚0.09402779835467559 1loss/FBt2/assignment_loss‚0.14949518442330048 /loss/FBt1/detection_loss‚2.8872874847480228 .loss/FBt3/detection_loss‚2.495476464430491 epoch‚1 _runtime‚2440.035247815 /loss/FRt2/detection_loss‚2.0240680991925974 0loss/SRqqt2/detection_loss‚2.318605193625326 0loss/FRt2/assignment_loss‚2.8341714455322786 2loss/SRqqt4/assignment_loss‚1.3974815445103412 2loss/SRqqt2/assignment_loss‚2.2896208983400594 0loss/FBt3/assignment_loss‚0.2875664291592723¢ù”Ï]Á× _step‚130 2loss/SRqqt3/assignment_loss‚1.4605585704458521 1loss/FBt4/assignment_loss‚0.04426601713794298 /loss/FRt4/detection_loss‚1.7176238317787647 1loss/SRqqt4/detection_loss‚1.0785805726604116 %loss/total_loss‚49.28623619632059 1loss/SRqqt1/assignment_loss‚2.297019791953704 .loss/FRt1/detection_loss‚2.847890059649944 .loss/FRt3/detection_loss‚3.141257127126058 /loss/FRt4/assignment_loss‚2.131458195845286 1loss/FBt1/assignment_loss‚0.09402779835467559 1loss/FBt2/assignment_loss‚0.14949518442330048 /loss/FBt1/detection_loss‚2.8872874847480228 .loss/FBt3/detection_loss‚2.495476464430491 epoch‚1 _runtime‚2440.035247815 /loss/FRt2/detection_loss‚2.0240680991925974 0loss/SRqqt2/detection_loss‚2.318605193625326 0loss/FRt2/assignment_loss‚2.8341714455322786 2loss/SRqqt4/assignment_loss‚1.3974815445103412 2loss/SRqqt2/assignment_loss‚2.2896208983400594 0loss/FBt3/assignment_loss‚0.2875664291592723 /loss/FRt3/assignment_loss‚4.869003895918528 -loss/FBt4/detection_loss‚3.08846802636981 trainer/global_step‚1599 /loss/FRt1/assignment_loss‚4.342697191238403 1loss/SRqqt1/detection_loss‚2.4010330193183003 1loss/SRqqt3/detection_loss‚2.1033736556134324 /loss/FBt2/detection_loss‚0.9852056989477853 % _timestamp‚1.7885376447243648e+09‚9ùK-“‚(:ˆ ±ÖëÔ¹ûúË gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.272' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2544 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾iŽoÂ(:ä ±ÖëÔ³­‚Ï disk.sda2.out‚0( disk.nvme0c0n1.out‚47730.66259765625 disk.sda3.out‚ 85.02734375 disk.dm-0.in‚ 5831.98828125# disk.dm-0.out‚47730.66259765625 disk.dm-1.out‚ 85.02734375% proc.memory.availableMB‚ 6.8203125 network.sent‚ 2.962311e+06% disk./.usageGB‚1179.1004753112793 disk.drbd1260.in‚0 disk.drbd1260.out‚0.359375 disk.sda3.in‚0 proc.cpu.threads‚17 network.recv‚815068# disk.nvme0c0n1.in‚ 5831.98828125! disk.nvme0n1.in‚ 5831.98828125 disk.sda.out‚ 85.02734375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.95837211608887) proc.memory.percent‚18.65997314453125) disk./.usagePercent‚79.13042222933491 disk.sda2.in‚0& disk.nvme0n1.out‚47730.66259765625 disk.sda.in‚0 disk.dm-1.in‚0 proc.memory.rssMB‚3057.25 cpu‚25.384192609306904>B­GÄ‚(:’ ÀÖëÔãþÌ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚175.771' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.154199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a«xgÅ‚(:Ü ÀÖëÔÊ¥ÒÏ disk.dm-0.in‚ 5997.21484375# disk.dm-0.out‚48100.42822265625 disk.sda2.out‚0 proc.memory.rssMB‚3057.25( disk.nvme0c0n1.out‚48100.42822265625 disk.sda.out‚ 85.359375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 cpu‚25.38277925588984 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 5997.21484375 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0) proc.memory.percent‚18.65997314453125 network.sent‚ 3.069951e+06 network.recv‚821145) disk./.usagePercent‚79.13039099641573& disk.nvme0n1.out‚48100.42822265625 disk.dm-1.out‚ 85.359375 disk.drbd1260.out‚ 0.36328125$ memory_percent‚99.95861053466797# proc.memory.availableMB‚6.78125$ disk./.usageGB‚1179.100009918213! disk.nvme0n1.in‚ 5997.21484375 disk.sda3.out‚ 85.359375 disk.sda.in‚0pç¬A’Æ‚(:‡ ÏÖëÔ«îÁË gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð;ÐrÇ‚(:ç ÏÖëÔ×­¨Ñ network.sent‚ 3.078031e+06 network.recv‚825997% disk./.usageGB‚1179.1000671386719 disk.drbd1260.out‚ 0.36328125 disk.sda3.out‚ 85.80078125 disk.sda1.out‚0 disk.dm-0.in‚ 5997.21484375$ memory_percent‚99.95853900909424 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚48278.01318359375 disk.sda1.in‚0 disk.dm-1.in‚0 proc.memory.rssMB‚3057.25 cpu‚25.381261845079813 disk.sda.out‚ 85.80078125 disk.dm-1.out‚ 85.80078125) proc.memory.percent‚18.65997314453125 proc.cpu.threads‚17) disk./.usagePercent‚79.13039483652874# disk.nvme0c0n1.in‚ 5997.21484375! disk.nvme0n1.in‚ 5997.21484375& disk.nvme0n1.out‚48278.01318359375 disk.sda3.in‚0 disk.sda.in‚0# disk.dm-0.out‚48278.01318359375& proc.memory.availableMB‚ 6.79296875ofmQÈ‚(:’ ÞÖëÔ¤™¼Î gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.079' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.615800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*”oÉ‚(:ä ÞÖëÔŠƒ˜Ô& disk.nvme0n1.out‚48498.77490234375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 5997.21484375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 86.33203125 proc.memory.rssMB‚3057.25# disk./.usageGB‚1179.07661819458 disk.sda2.in‚0 disk.sda3.out‚ 86.33203125$ memory_percent‚99.96011257171631& proc.memory.availableMB‚ 6.53515625 cpu‚25.379639011901553 network.recv‚829458 disk.drbd1260.out‚ 0.3671875# disk.nvme0c0n1.in‚ 5997.21484375( disk.nvme0c0n1.out‚48498.77490234375 disk.sda.out‚ 86.33203125# disk.dm-0.out‚48498.77490234375 disk.dm-1.in‚0) proc.memory.percent‚18.65997314453125 network.sent‚ 3.081186e+06) disk./.usagePercent‚79.12882115821577! disk.nvme0n1.in‚ 5997.21484375&å¥2Ê- _wandbruntime‚2500  _runtime‚2500OäÆËÀ & lr-AdamW/pg2‚0.0016787014376692993 trainer/global_step‚1649 % _timestamp‚1.7885377048066742e+09 _runtime‚ 2500.11785526 _step‚131 & lr-AdamW/pg1‚0.0016787014376692993ŒŸéïËÌÅ % _timestamp‚1.7885377048066742e+09 _runtime‚ 2500.11785526 _step‚131 & lr-AdamW/pg1‚0.0016787014376692993 & lr-AdamW/pg2‚0.0016787014376692993 trainer/global_step‚1649ƒÝ$÷¿2Í- _wandbruntime‚2500  _runtime‚2500lÍPo'ÎåO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41628481 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138113767550464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139995163276288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1882305696256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20516706 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1881395725824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34249782 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154407209169408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34249782 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚1649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _step‚132 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75904605 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138113785553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34249780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚133577727279376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1875264675437 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1881395725824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154407459795456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138113767550464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1969177189376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41654823 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30508822 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9992741 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152438258665984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1968950503424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152438282606080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139995163276288 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_Xáâä|bytes.all.allocated‚135452991954813 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640æê°·ð'ÏêO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139996091249664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20516704 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139996091249664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135452991954813 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41628481 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138113767550464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139995163276288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1882305696256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20516706 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1881395725824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34249782 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚154407209169408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34249782 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚1649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 _step‚132 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75904605 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138113785553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34249780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚133577727279376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1875264675437 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1881395725824 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154407459795456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138113767550464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1969177189376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41654823 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30508822 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9992741 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚152438258665984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1968950503424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152438282606080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139995163276288 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138195404639232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154498603369472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154498860273664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970118999040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75949622 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140077903822336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9998522 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 $ _timestamp‚1.788537706031069e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133656727834560 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30526651 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41653312 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135533126822843 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1876381948923 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75949622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41679680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41679680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256904192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232964096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34269942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1882517186048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20528771 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20528769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140078831809024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140078831809024 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75949622 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140077903822336 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9998522 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 $ _timestamp‚1.788537706031069e+09 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚133656727834560 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30526651 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41653312 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135533126822843 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1876381948923 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75949622 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41679680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41679680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256904192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232964096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34269942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1882517186048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20528771 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20528769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140078831809024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140078831809024 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138195404639232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154498603369472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154498860273664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1970118999040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0…†pý2Ó- _wandbruntime‚2501  _runtime‚2501Yk UÔÏ 0loss/FBt2/assignment_loss‚0.2894809104429537 .loss/FRt4/detection_loss‚1.664706542143332 trainer/global_step‚1649 /loss/FRt1/assignment_loss‚5.086051147431135 1loss/SRqqt3/detection_loss‚2.2795406172915205 epoch‚1 0loss/FRt2/assignment_loss‚2.4202457749181323 0loss/SRqqt1/assignment_loss‚2.04470455204999 2loss/SRqqt4/assignment_loss‚1.3404811348866772 /loss/FBt4/assignment_loss‚0.283677787904162 1loss/SRqqt1/detection_loss‚2.6499424941009946 1loss/SRqqt4/detection_loss‚1.2390067874042838 /loss/FBt2/detection_loss‚0.8105467750972725 1loss/SRqqt2/assignment_loss‚1.670074301687154 0loss/FBt3/assignment_loss‚0.4840585639861632 /loss/FBt3/detection_loss‚2.4228723365813494 .loss/FBt4/detection_loss‚2.966458249092102 $ _timestamp‚1.788537706034278e+09 0loss/SRqqt2/detection_loss‚2.799153447151184 .loss/FBt1/detection_loss‚5.130559308962389 2loss/SRqqt3/assignment_loss‚1.9960533067947481 /loss/FRt1/detection_loss‚2.9103991547599435 /loss/FRt2/detection_loss‚2.2486028584341207 1loss/FBt1/assignment_loss‚0.44959859956394543 _step‚134 /loss/FRt3/detection_loss‚2.7485640707768892 %loss/total_loss‚52.78616233936171 _runtime‚2501.345215553 /loss/FRt3/assignment_loss‚4.438552906638698 0loss/FRt4/assignment_loss‚2.4128307112625667$Uî´ÕÔ 1loss/SRqqt2/assignment_loss‚1.670074301687154 0loss/FBt3/assignment_loss‚0.4840585639861632 /loss/FBt3/detection_loss‚2.4228723365813494 .loss/FBt4/detection_loss‚2.966458249092102 $ _timestamp‚1.788537706034278e+09 0loss/SRqqt2/detection_loss‚2.799153447151184 .loss/FBt1/detection_loss‚5.130559308962389 2loss/SRqqt3/assignment_loss‚1.9960533067947481 /loss/FRt1/detection_loss‚2.9103991547599435 /loss/FRt2/detection_loss‚2.2486028584341207 1loss/FBt1/assignment_loss‚0.44959859956394543 _step‚134 /loss/FRt3/detection_loss‚2.7485640707768892 %loss/total_loss‚52.78616233936171 _runtime‚2501.345215553 /loss/FRt3/assignment_loss‚4.438552906638698 0loss/FRt4/assignment_loss‚2.4128307112625667 0loss/FBt2/assignment_loss‚0.2894809104429537 .loss/FRt4/detection_loss‚1.664706542143332 trainer/global_step‚1649 /loss/FRt1/assignment_loss‚5.08605114743113|gyö¦5 1loss/SRqqt3/detection_loss‚2.2795406172915205 epoch‚1 0loss/FRt2/assignment_loss‚2.4202457749181323 0loss/SRqqt1/assignment_loss‚2.04470455204999 2loss/SRqqt4/assignment_loss‚1.3404811348866772 /loss/FBt4/assignment_loss‚0.283677787904162 1loss/SRqqt1/detection_loss‚2.6499424941009946 1loss/SRqqt4/detection_loss‚1.2390067874042838 /loss/FBt2/detection_loss‚0.8105467750972725†U€ŽÖ‚(:ƒ íÖëÔÑûµË gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùòªmׂ(:â íÖëÔâÛ«Ï disk.dm-1.out‚ 86.92578125) proc.memory.percent‚18.65997314453125 cpu‚25.37819305710748 proc.cpu.threads‚17 network.sent‚ 3.086382e+06 disk.sda2.out‚0 disk.drbd1260.out‚ 0.3671875! disk.nvme0n1.in‚ 5997.21484375& disk.nvme0n1.out‚48772.53662109375# disk.dm-0.out‚48772.53662109375$ memory_percent‚99.95899200439453# proc.memory.availableMB‚6.71875 network.recv‚832902 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 5997.21484375 proc.memory.rssMB‚3057.25% disk./.usageGB‚1179.1010665893555 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 5997.21484375( disk.nvme0c0n1.out‚48772.53662109375 disk.sda3.out‚ 86.92578125 disk.sda.out‚ 86.92578125 disk.sda1.out‚0 disk.dm-1.in‚0) disk./.usagePercent‚79.13046191050272 disk.sda2.in‚0 disk.sda.in‚0ø{±“Ø‚(:ˆ üÖëÔÛÊ“Ë gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØL¢]Ù‚(:Ò üÖëÔ¶‘ôÍ disk.dm-1.out‚ 87.203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0% proc.memory.availableMB‚ 6.2109375 disk.sda1.in‚0 disk.dm-0.out‚49032.46484375 disk.dm-1.in‚0% disk.nvme0c0n1.out‚49032.46484375 disk.nvme0n1.in‚ 6132.5078125 disk.sda3.out‚ 87.203125 disk.sda.in‚0 disk.sda.out‚ 87.203125 network.sent‚ 3.196619e+06 disk.sda1.out‚0 disk.drbd1260.out‚ 0.37109375# disk.nvme0n1.out‚49032.46484375$ memory_percent‚99.96209144592285) proc.memory.percent‚18.65997314453125 cpu‚25.37650087346493 proc.cpu.threads‚17 network.recv‚839073 disk.dm-0.in‚ 6132.5078125" disk.nvme0c0n1.in‚ 6132.5078125 disk.sda3.in‚0 proc.memory.rssMB‚3057.25( disk./.usagePercent‚79.1303989326493% disk./.usageGB‚1179.1001281738281¨Ç “Ú‚(:ˆ ‹×ëÔ¹€ÝË gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š’¿ŠnÛ‚(:ã ‹×ëÔÛýÎ network.sent‚ 3.200899e+06 network.recv‚843723 disk.sda3.out‚ 87.66015625 disk.sda1.in‚0# disk.dm-0.out‚49185.97119140625 disk.sda2.out‚0 disk.drbd1260.out‚ 0.37109375" disk.nvme0c0n1.in‚ 6132.5078125 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚49185.97119140625& disk.nvme0n1.out‚49185.97119140625& proc.memory.availableMB‚ 6.45703125 disk.dm-1.out‚ 87.66015625% disk./.usageGB‚1179.1001281738281 disk.sda3.in‚0 disk.dm-0.in‚ 6132.5078125 disk.nvme0n1.in‚ 6132.5078125$ memory_percent‚99.96058940887451) proc.memory.percent‚18.65997314453125 cpu‚25.375067357890828( disk./.usagePercent‚79.1303989326493 disk.sda.out‚ 87.66015625 disk.dm-1.in‚0 proc.memory.rssMB‚3057.25 proc.cpu.threads‚17ÕU² “Ü‚(:ˆ š×ëÔ—þ¸Ë gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û”yñgÝ‚(:Ü š×ëÔÒ³ÝÍ disk.dm-1.in‚0 disk.dm-1.out‚ 88.2109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚99.96058940887451) proc.memory.percent‚18.65997314453125 cpu‚25.373504663606084) disk./.usagePercent‚79.12725029598568 disk.nvme0n1.in‚ 6132.5078125 disk.sda3.out‚ 88.2109375 disk.sda.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 6.45703125 proc.memory.rssMB‚3057.25 proc.cpu.threads‚17 network.sent‚ 3.205877e+06 network.recv‚847205% disk./.usageGB‚1179.0532112121582 disk.drbd1260.out‚0.375& disk.nvme0n1.out‚49503.30517578125 disk.sda3.in‚0 disk.dm-0.in‚ 6132.5078125# disk.dm-0.out‚49503.30517578125 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 6132.5078125( disk.nvme0c0n1.out‚49503.30517578125 disk.sda.out‚ 88.2109375D&Öß2Þ- _wandbruntime‚2561  _runtime‚2561îƒmÅß¿ & lr-AdamW/pg2‚0.0016770729940236886 trainer/global_step‚1699 $ _timestamp‚1.788537766162414e+09 _runtime‚ 2561.47361309 _step‚135 & lr-AdamW/pg1‚0.0016770729940236886?/ôˆÊàÄ _step‚135 & lr-AdamW/pg1‚0.0016770729940236886 & lr-AdamW/pg2‚0.0016770729940236886 trainer/global_step‚1699 $ _timestamp‚1.788537766162414e+09 _runtime‚ 2561.47361309‡gèê2á- _wandbruntime‚2561  _runtime‚2561œ2Wòî'âèO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144133132965888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 trainer/global_step‚1699 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1937483150848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78131163 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42899725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1938393121280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144132204992512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10284526 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139459748937121 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35257782 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156949567828480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137528589967216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1937483150848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚137528607006576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21119954 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158977015410688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚2561.475170135 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144132204992512 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144133132965888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35257780 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1938393121280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10283911 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2027704499712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78157507 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885377661639822e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142194721841664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31403865 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156949543888384 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142194739844608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139458823922765 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42873383 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35257780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142194721841664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1931141930545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142194739844608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2027471522304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21119956 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158977272328192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0èWó'ãíO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144133132965888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 trainer/global_step‚1699 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1937483150848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78131163 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42899725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1938393121280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144132204992512 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10284526 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232977408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139459748937121 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35257782 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156949567828480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚137528589967216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1937483150848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚137528607006576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21119954 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158977015410688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚2561.475170135 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144132204992512 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144133132965888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35257780 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1938393121280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10283911 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2027704499712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256917504 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78157507 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885377661639822e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142194721841664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31403865 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156949543888384 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142194739844608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139458823922765 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42873383 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35257780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142194721841664 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1931141930545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142194739844608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2027471522304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21119956 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158977272328192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432ˆ‹máÿ2ä- _wandbruntime‚2562  _runtime‚2562¾Ṯ÷åòM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142276358930432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1939514594816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137607607561760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157039793532928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2562.508877851 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139538958790587 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78202524 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2028864476672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21132019 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78176154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocateGH¶›d‚10290175 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21132021 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1938604611072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚137 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144215873525248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142276358930432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885377671974504e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35277942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142276340927488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1931351228827 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1932259204031 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42924582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31421576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142276340927488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42898214 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42924582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640÷ –ý&æ÷M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31421576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142276340927488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42898214 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42924582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142276358930432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1939514594816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚137607607561760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157039793532928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2562.508877851 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139538958790587 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78202524 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2028864476672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21132019 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78176154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10290175 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21132021 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1938604611072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚137 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144215873525248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142276358930432 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885377671974504e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35277942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142276340927488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1931351228827 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709888 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1932259204031 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42924582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B–žð‚(:“ Ö×ëÔÄšÛË gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.635' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.526999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¸Rmñ‚(:â Ö×ëÔÒ‰¯Î cpu‚25.36797705572832 proc.cpu.threads‚17 network.recv‚865546 disk.drbd1260.out‚ 0.3828125! disk.nvme0n1.in‚ 6273.48828125 disk.sda.out‚ 90.66796875# disk.dm-0.out‚50448.06591796875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6273.48828125& disk.nvme0n1.out‚50448.94091796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 6273.48828125) proc.memory.percent‚18.65997314453125) disk./.usagePercent‚79.08377304846071% disk./.usageGB‚1178.4053688049316 disk.sda2.out‚0( disk.nvme0c0n1.out‚50448.94091796875 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 90.66796875 network.sent‚ 3.332918e+06 disk.drbd1260.in‚0 disk.sda3.out‚ 90.66796875$ memory_percent‚99.96089935302734# proc.memory.availableMB‚6.40625 proc.memory.rssMB‚3057.25¸Ç¿2ò- _wandbruntime‚2622  _runtime‚2622#YØÆóÀ trainer/global_step‚1749 $ _timestamp‚1.788537827455412e+09 _runtime‚2622.766376228 _step‚139 & lr-AdamW/pg1‚0.0016752706779554905 & lr-AdamW/pg2‚0.0016752706779554905ñr¶»ËôÅ _step‚139 & lr-AdamW/pg1‚0.0016752706779554905 & lr-AdamW/pg2‚0.0016752706779554905 trainer/global_step‚1749 $ _timestamp‚1.788537827455412e+09 _runtime‚2622.766376228‹CL­¥2õ- _wandbruntime‚2622  _runtime‚2622é¹»|í'öçO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161460829110784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44144627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143466505919429 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141479469694416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32297321 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148270174682112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21723204 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10574723 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1986111210657 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2086364457984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148269246708736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2086150354944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163546979465728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161460853050880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 trainer/global_step‚1749 _runtime‚ 2622.76765582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146275694135808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32297929 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148269246708736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146275676132864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1993570575872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10574117 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163547217508864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1993570575872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146275694135808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36265780 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44118285 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1994480546304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36265780 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136-Yž÷ìO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21723206 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.lar̙֗T'ge_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141479486733776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36265782 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143465580905073 % _timestamp‚1.7885378274565656e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44144627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148270174682112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21723204 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10574723 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1986111210657 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2086364457984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148269246708736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2086150354944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163546979465728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161460853050880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 trainer/global_step‚1749 _runtime‚ 2622.76765582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146275694135808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32297929 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148269246708736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146275676132864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1993570575872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10574117 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163547217508864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1993570575872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146275694135808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36265780 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44118285 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1994480546304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36265780 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161460829110784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44144627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143466505919429 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141479469694416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32297321 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80455426 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80429056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21735269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161551054815232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1987228483935 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146357295218688 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143546640787459 trainer/global_step‚1749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44169484 % _timestamp‚1.7885378284871113e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1988136459139 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2087304617984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36285942 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36285940 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36285940 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143545715772895 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146357295218688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1995602019840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10579876 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1994692036096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163638359433216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _runtime‚2623.799163152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1995602019840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141558504328320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148352915241472 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44143116 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146357313221632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1994692036096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259001344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80455426 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2087539679232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80429056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235061248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32315766 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36285942 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163638618434560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32315145 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21735271 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688‡ ý&ú÷M TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2087304617984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36285942 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36285940 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36285940 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143545715772895 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146357295218688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1995602019840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10579876 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1994692036096 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163638359433216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 _runtime‚2623.799163152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1995602019840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141558504328320 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148352915241472 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44143116 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146357313221632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1994692036096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259001344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80455426 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2087539679232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80429056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235061248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32315766 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36285942 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163638618434560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32315145 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21735271 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146357313221632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80455426 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80429056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21735269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161551054815232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1987228483935 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146357295218688 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143546640787459 trainer/global_step‚1749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44169484 % _timestamp‚1.7885378284871113e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1988136459139 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640eă2û- _wandbruntime‚2623  _runtime‚2623ª *lYüÓ 1loss/SRqqt4/detection_loss‚0.8644321326933363 /loss/FBt1/detection_loss‚3.3642752445661106 %loss/total_loss‚51.87308459136972 _step‚142 2loss/SRqqt3/assignment_loss‚1.9354364226971355 .loss/FRt4/detection_loss‚1.934745175709181 0loss/FRt4/assignment_loss‚2.5342103654847428 1loss/SRqqt2/assignment_loss‚2.217210954741428 1loss/FBt2/assignment_loss‚0.31157051718407053 1loss/FBt3/assignment_loss‚0.30305377780552306 1loss/SRqqt3/detection_loss‚2.4688162480081832 /loss/FBt2/detection_loss‚0.8301386684934432 trainer/global_step‚1749 epoch‚1 % _timestamp‚1.7885378284909122e+09 /loss/FRt3/assignment_loss‚4.571020342054821 0loss/FBt1/assignment_loss‚0.5469584477496727 1loss/FBt4/assignment_loss‚0.25492150088151294 .loss/FRt2/detection_loss‚2.040994784484307 .loss/FRt3/detection_loss‚3.541372883887518 1loss/SRqqt2/detection_loss‚2.7614228035274304 _runtime‚2623.802348838 2loss/SRqqt1/assignment_loss‚2.2468663133956768 2loss/SRqqt4/assignment_loss‚1.3578320010017384 /loss/FBt3/detection_loss‚2.0183313405141234 .loss/FBt4/detection_loss‚3.329191896650526 .loss/FRt1/assignment_loss‚4.40555768609047 1loss/SRqqt1/detection_loss‚2.7243579868917114 /loss/FRt2/assignment_loss‚2.642574989133411 /loss/FRt1/detection_loss‚2.6677921077236535©3aÌÉýØ 2loss/SRqqt4/assignment_loss‚1.3578320010017384 /loss/FBt3/detection_loss‚2.0183313405141234 .loss/FBt4/detection_loss‚3.329191896650526 .loss/FRt1/assignment_loss‚4.40555768609047 1loss/SRqqt1/detection_loss‚2.7243579868917114 /loss/FRt2/assignment_loss‚2.642574989133411 /loss/FRt1/detection_loss‚2.6677921077236535 1loss/SRqqt4/detection_loss‚0.8644321326933363 /loss/FBt1/detection_loss‚3.3642752445661106 %loss/total_loss‚51.87308459136972 _step‚142 2loss/SRqqt3/assignment_loss‚1.9354364226971355 .loss/FRt4/detection_loss‚1.934745175709181 0loss/FRt4/assignment_loss‚2.5342103654847428 1loss/SRqqt2/assignment_loss‚2.217210954741428 1loss/FBt2/assignment_loss‚0.31157051718407053 1loss/FBt3/assignment_loss‚0.30305377780552306 1loss/SRqqt3/detection_loss‚2.4688162480081832 /loss/FBt2/detection_loss‚0.8301386684934432 trainer/global_step‚1749 epoch‚1 % _timestamp‚1.7885378284909122e+09 /loss/FRt3/assignment_loss‚4.571020342054821 0loss/FBt1/assignment_loss‚0.5469584477496727 1loss/FBt4/assignment_loss‚0.25492150088151294 .loss/FRt2/detection_loss‚2.040994784484307 .loss/FRt3/detection_loss‚3.541w€*ð•372883887518 1loss/SRqqt2/detection_loss‚2.7614228035274304 _runtime‚2623.802348838 2loss/SRqqt1/assignment_loss‚2.2468663133956768Ž¾bCq“þ‚(:ˆ å×ëÔ‘ËåÉ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô”)mÿ‚(:â å×ëÔíțΠdisk.drbd1260.in‚0( disk.nvme0c0n1.out‚50774.81591796875 disk.sda3.out‚ 91.03515625 disk.sda.in‚0 disk.dm-0.in‚ 6273.48828125 disk.dm-1.in‚0 disk.dm-1.out‚ 91.03515625) proc.memory.percent‚18.65997314453125 network.sent‚ 3.349145e+06# disk.nvme0c0n1.in‚ 6273.48828125! disk.nvme0n1.in‚ 6273.48828125 disk.sda3.in‚0 disk.sda.out‚ 91.03515625 disk.sda1.out‚0 disk.sda2.out‚0# memory_percent‚99.9638557434082) disk./.usagePercent‚79.08377765659634% disk./.usageGB‚1178.4054374694824 disk.sda1.in‚0 proc.memory.rssMB‚3057.25 network.recv‚868794 disk.drbd1260.out‚ 0.3828125& disk.nvme0n1.out‚50774.81591796875# disk.dm-0.out‚50774.81591796875 disk.sda2.in‚0$ proc.memory.availableMB‚5.921875 cpu‚25.36647626201621 proc.cpu.threads‚170ÎJú“€‚(:ˆ ô×ëÔõŸúË gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.573' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*üµ¾k‚(:à ô×ëÔ†£Î disk.sda.out‚ 91.4453125 disk.dm-1.out‚ 91.4453125 proc.memory.rssMB‚3057.25 proc.cpu.threads‚17) disk./.usagePercent‚79.08377381648332% disk./.usageGB‚1178.4053802490234 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6273.48828125( disk.nvme0c0n1.out‚51034.35595703125! disk.nvme0n1.in‚ 6273.48828125 disk.sda.in‚0 network.sent‚ 3.444603e+06 network.recv‚875164 disk.drbd1260.out‚ 0.38671875 disk.sda3.in‚0 disk.sda3.out‚ 91.4453125 disk.sda1.in‚0 disk.dm-0.in‚ 6273.48828125# disk.dm-0.out‚51034.35595703125 disk.sda1.out‚0 disk.dm-1.in‚0# memory_percent‚99.9598503112793$ proc.memory.availableMB‚6.578125) proc.memory.percent‚18.65997314453125 cpu‚25.36508102383864 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚51034.35595703125ŠS'“‚‚(:ˆ ƒØëÔèÅ’Ê gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.417' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕŒ€/`ƒ‚(:Õ ƒØëÔ£ÿãÌ cpu‚25.363547255487042 disk.sda2.in‚0 disk.drbd1260.out‚ 0.38671875( disk.nvme0c0n1.out‚51543.52392578125 disk.sda3.out‚ 91.8046875$ memory_percent‚99.96137619018555 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 6410.1875 disk.nvme0n1.in‚ 6410.1875& disk.nvme0n1.out‚51543.52392578125 disk.sda1.in‚0# disk.dm-0.out‚51543.52392578125) proc.memory.percent‚18.65997314453125 network.sent‚ 3.450823e+06 network.recv‚879900 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 91.8046875 disk.sda1.out‚0 disk.dm-0.in‚ 6410.1875 disk.dm-1.out‚ 91.8046875( disk./.usagePercent‚79.0837766325662% disk./.usageGB‚1178.4054222106934 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0$ proc.memory.availableMB‚6.328125 proc.memory.rssMB‚3057.25{ŒÜ“„‚(:ˆ ’ØëÔ¾»âÊ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.789' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àƒsûd…‚(:Ù ’ØëÔÆœºÌ) disk./.usagePercent‚79.08377560853606 disk.sda.in‚0 disk.dm-0.in‚ 6410.1875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0& proc.memory.availableMB‚ 6.08203125 disk.sda.out‚ 92.46484375 disk.sda1.out‚0 network.sent‚ 3.455838e+06% disk./.usageGB‚1178.4054069519043 disk.nvme0c0n1.in‚ 6410.1875( disk.nvme0c0n1.out‚51658.88623046875 disk.nvme0n1.in‚ 6410.1875 disk.sda3.out‚ 92.46484375 network.recv‚883276 disk.sda1.in‚0# disk.dm-0.out‚51658.88623046875 disk.dm-1.out‚ 92.46484375 disk.drbd1260.out‚0.390625 proc.memory.rssMB‚3057.25 cpu‚25.362191557970352& disk.nvme0n1.out‚51658.88623046875 disk.sda3.in‚0 disk.sda2.out‚0$ memory_percent‚99.96287822723389) proc.memory.percent‚18.65997314453125 proc.cpu.threads‚17Àù‡t2†- _wandbruntime‚2681  _runtime‚26819:’¤Æ‡À _runtime‚2681.792345471 _step‚143 & lr-AdamW/pg1‚0.0016732948650646719 & lr-AdamW/pg2‚0.0016732948650646719 trainer/global_step‚1799 $ _timestamp‚1.788537886481208e+09pñ«¡ËˆÅ & lr-AdamW/pg2‚0.0016732948650646719 trainer/global_step‚1799 $ _timestamp‚1.788537886481208e+09 _runtime‚2681.792345471 _step‚143 & lr-AdamW/pg1‚0.0016732948650646719%U Ï2‰- _wandbruntime‚2681  _runtime‚2681•íåî'ŠèO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _step‚144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37273780 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152407216398336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2050567971328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2049658000896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33193132 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚2681.794029235 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45389529 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2041988465765 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885378864827294e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2050567971328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37273780 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10867315 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚147472337887381 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152406288424960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2145122685440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165972114333184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145430366460976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45363187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152407216398336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150356630424064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145430349421616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45363187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82663311 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚1799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2145351468544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152406288424960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150356630424064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚147473262901737 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10866678 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37273782 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82636967 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37273782 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2049658000896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168117489741824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33193771 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2042896440761 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45389529 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22326456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150356648427008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150356648427008 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82636967 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37273782 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2049658000896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168117489741824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33193771 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2042896440761 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45389529 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22326456 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150356648427008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚150356648427008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45389529 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2041988465765 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 % _timestamp‚1.7885378864827294e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2050567971328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37273780 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10867315 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚147472337887381 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152406288424960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2145122685440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165972114333184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145430366460976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45363187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152407216398336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150356630424064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145430349421616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45363187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82663311 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 trainer/global_step‚1799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2145351468544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152406288424960 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150356630424064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚147473262901737 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10866678 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37273782 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513Ûa42Œ- _wandbruntime‚2683  _runtime‚2683²Ú½ðñM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168208881540608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚147553397769767 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82681958 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37293940 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45388018 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82708328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152489956957696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37293942 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33210910 % _timestamp‚1.7885378878778913e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152489956957696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2051689444864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82708328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _runtime‚ 2683.19011413 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33211552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259001344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168208622539264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚1799 _step‚145 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37293942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145509367016160 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2050779461120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22338521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2146517562880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150438249509888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152489028971008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2146282501632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145509384055520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166062340037632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OGDeviceSt'4tÐó atsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2043105739043 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37293940 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2043105739043 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37293940 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33210910 % _timestamp‚1.7885378878778913e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152489956957696 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2051689444864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82708328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _runtime‚ 2683.19011413 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33211552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259001344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168208622539264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 trainer/global_step‚1799 _step‚145 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37293942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145509367016160 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2050779461120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22338521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2146517562880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150438249509888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152489028971008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2146282501632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0‘m¬ø~2- _wandbruntime‚2683  _runtime‚2683M$ìöTÎ .loss/FRt3/detection_loss‚2.835202968120575 0loss/SRqqt4/detection_loss‚1.353768221249706 .loss/FBt4/detection_loss‚2.952691205910274 $ _timestamp‚1.788537887881687e+09 /loss/FBt3/detection_loss‚2.8024493977427483 /loss/FRt4/assignment_loss‚2.439047644012853 0loss/SRqqt1/assignment_loss‚2.46297906580809 2loss/SRqqt2/assignment_loss‚2.2934166824116424 1loss/SRqqt3/assignment_loss‚2.088453468951312 2loss/SRqqt4/assignment_loss‚1.0368072545646052 1loss/FBt2/assignment_loss‚0.28943199482847054 0loss/SRqqt1/detection_loss‚2.201230824799151 &loss/total_loss‚50.567627967464745 1loss/FBt1/assignment_loss‚0.14808227154963577 /loss/FRt4/detection_loss‚1.8274827828924907 0loss/SRqqt2/detection_loss‚2.572986301253824 /loss/FBt1/detection_loss‚3.3335705124414883 _runtime‚2683.192630474 /loss/FBt2/detection_loss‚0.8136515224813133 _step‚146 1loss/SRqqt3/detection_loss‚2.3383549280238873 /loss/FRt1/assignment_loss‚4.100440204143524 /loss/FRt3/assignment_loss‚3.775086724758148 0loss/FBt3/assignment_loss‚0.6556685946560687 0loss/FBt4/assignment_loss‚0.9975339068997917 epoch‚1 trainer/global_step‚1799 /loss/FRt1/detection_loss‚2.8518956918269396 /loss/FRt2/assignment_loss‚2.232425466889427 .loss/FRt2/detection_loss‚2.164970331248783BÒçY‘Ó epoch‚1 trainer/global_step‚1799 /loss/FRt1/detection_loss‚2.8518956918269396 /loss/FRt2/assignment_loss‚2.232425466889427 .loss/FRt2/detection_loss‚2.164970331248783 .loss/FRt3/detection_loss‚2.835202968120575 0loss/SRqqt4/detection_loss‚1.353768221249706 .loss/FBt4/detection_loss‚2.952691205910274 $ _timestamp‚1.788537887881687e+09 /loss/FBt3/detection_loss‚2.8024493977427483 /loss/FRt4/assignment_loss‚2.439047644012853 0loss/SRqqt1/assignment_loss‚2.46297906580809 2loss/SRqqt2/assignment_loss‚2.2934166824116424 1loss/SRqqt3/assignment_loss‚2.088453468951312 2loss/SRqqt4/assignment_loss‚1.0368072545646052 1loss/FBt2/assignment_loss‚0.28943199482847054 0loss/SRqqt1/detection_loss‚2.201230824799151 &loss/total_loss‚50.567627967464745 1loss/FBt1/assignment_loss‚0.14808227154963577 /loss/FRt4/detection_loss‚1.8274827828924907 0loss/SRqqt2/detection_loss‚2.572986301253824 /loss/FBt1/detection_loss‚3.3335705124414883 _runtime‚2683.192630474 /loss/FBt2/detection_loss‚0.8136515224813133 _step‚146 1loss/SRqqt3/detection_loss‚2.3383549280238873 /loss/FRt1/assignment_loss‚4.100440204143524 /loss/FRt3/assignment_loss‚3.775086724758148 0loss/FBt3/assignment_loss‚0.6556685946560687 0loss/FBt4/assignment_loss‚0.9975339068997917’h–“’‚(:ˆ ¡ØëÔ쵉Í gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚153.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|wb“‚(:× ¡ØëÔ°»ÞÎ disk.nvme0n1.in‚ 6410.1875 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 3.461055e+06# disk.dm-0.out‚51659.15966796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚0.390625 disk.sda3.out‚ 92.83984375$ memory_percent‚99.96285438537598$ disk./.usageGB‚1178.405387878418 disk.dm-0.in‚ 6410.1875& disk.nvme0n1.out‚51659.15966796875 disk.sda3.in‚0 proc.cpu.threads‚17 network.recv‚887123 disk.nvme0c0n1.in‚ 6410.1875 disk.sda.out‚ 92.83984375 disk.sda1.in‚0% proc.memory.availableMB‚ 6.0859375 proc.memory.rssMB‚3057.25) proc.memory.percent‚18.65997314453125 cpu‚25.360800279650288) disk./.usagePercent‚79.08377432849838 disk.dm-1.in‚0 disk.dm-1.out‚ 92.83984375 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚51659.15966796875ÖÛû\”‚(:Ñ °ØëÔÂø¿Ë network.recv‚893378 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 6410.1875' disk.nvme0c0n1.out‚51996.0869140625 disk.dm-0.in‚ 6410.1875" disk.dm-0.out‚51996.0869140625 proc.memory.rssMB‚3057.25( disk./.usagePercent‚79.0750355113171 disk.nvme0n1.in‚ 6410.1875 disk.sda.out‚ 93.2109375 disk.sda1.out‚0 disk.dm-1.in‚0) proc.memory.percent‚18.65997314453125 cpu‚25.35952952096947% disk.nvme0n1.out‚51996.0869140625 disk.sda3.out‚ 93.2109375 network.sent‚ 3.571346e+06% disk./.usageGB‚1178.2751731872559 disk.drbd1260.out‚ 0.39453125# memory_percent‚99.9631404876709% proc.memory.availableMB‚ 6.0390625 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 93.2109375ryÉðž•‚(:“ °ØëÔû–ÝË gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.019' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.403800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´"Ör“–‚(:ˆ ¿ØëÔÓ´ÀÊ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.972' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ™e—‚(:Ú ¿ØëԥȌΠdisk.nvme0n1.in‚ 6543.6015625 disk.sda.in‚0$ memory_percent‚99.96159076690674& proc.memory.availableMB‚ 6.29296875) proc.memory.percent‚18.65997314453125 cpu‚25.358251944709693$ disk./.usageGB‚1178.297550201416 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 6543.6015625% disk.nvme0n1.out‚52279.2119140625 disk.sda3.out‚ 94.046875 disk.dm-0.in‚ 6543.6015625 disk.dm-1.out‚ 94.046875 network.sent‚ 3.575668e+06 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.39453125' disk.nvme0c0n1.out‚52279.2119140625 disk.sda3.in‚0 disk.sda.out‚ 94.046875 disk.sda1.in‚0 network.recv‚897993) disk./.usagePercent‚79.07653725151295 disk.sda1.out‚0" disk.dm-0.out‚52279.2119140625 disk.dm-1.in‚0 proc.memory.rssMB‚3057.25 proc.cpu.threads‚17Wñ@¢œ˜‚(:‘ ÎØëÔ†ó…Ê gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.9' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.780000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÜËj™‚(:ß ÎØëÔ¡»ŸÌ disk.sda2.out‚0& disk.nvme0n1.out‚52692.47412109375 disk.sda.out‚ 94.62109375 disk.sda1.in‚0$ memory_percent‚99.96166229248047# proc.memory.availableMB‚6.28125( disk.nvme0c0n1.out‚52692.47412109375 disk.sda3.in‚0 disk.sda3.out‚ 94.62109375 disk.dm-0.in‚ 6543.6015625# disk.dm-0.out‚52692.47412109375 proc.memory.rssMB‚3057.25 proc.cpu.threads‚17 network.sent‚ 3.580717e+06 network.recv‚901186 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 6543.6015625 disk.sda1.out‚0 disk.dm-1.in‚0) proc.memory.percent‚18.65997314453125 cpu‚25.35680286275207% disk./.usageGB‚1178.2741012573242 disk.drbd1260.out‚ 0.3984375" disk.nvme0c0n1.in‚ 6543.6015625 disk.sda.in‚0 disk.dm-1.out‚ 94.62109375) disk./.usagePercent‚79.07496357319997 disk.sda2.in‚0j®Ø¦2š- _wandbruntime‚2742  _runtime‚2742-É+‰Ç›Á trainer/global_step‚1849 % _timestamp‚1.7885379473035448e+09 _runtime‚2742.614577325 _step‚147 & lr-AdamW/pg1‚0.0016711459671076782 & lr-AdamW/pg2‚0.0016711459671076782 û…ÌœÆ trainer/global_step‚1849 % _timestamp‚1.7885379473035448e+09 _runtime‚2742.614577325 _step‚147 & lr-AdamW/pg1‚0.0016711459671076782 & lr-AdamW/pg2‚0.0016711459671076782“‘o²¼2- _wandbruntime‚2742  _runtime‚27420ÒòKí'žçO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2097865720873 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2105745425920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2204109790720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38281782 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11158484 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172687284757504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46634431 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11157869 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156543330141184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2106655396352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151479094869689 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156544258114560 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚148 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38281780 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149381246188176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34087573 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149381229148816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156543330141184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170483423495680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2098773695869 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46608089 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84916213 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46608089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34088190 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788537947304981e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46634431 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38281782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170483399555584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154437584715264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22929706 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2106655396352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚2742.616144199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22929704 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154437584715264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84889869 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172687533286400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151480019884045™r»ÜŸìO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2106655396352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 9250143¯o¾&56 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚2742.616144199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22929704 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154437584715264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84889869 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172687533286400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚151480019884045 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2097865720873 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2105745425920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2204109790720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38281782 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11158484 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172687284757504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46634431 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11157869 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156543330141184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2106655396352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151479094869689 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156544258114560 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚148 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38281780 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149381246188176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34087573 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149381229148816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156543330141184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170483423495680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2098773695869 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46608089 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84916213 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46608089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34088190 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 $ _timestamp‚1.788537947304981e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46634431 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38281782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170483399555584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154437584715264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22929706 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617”ò/¢2 - _wandbruntime‚2743  _runtime‚2743³ø&¡òM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚149 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2205069614592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚151559229737511 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34105366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84934860 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38301940 % _timestamp‚1.7885379484296093e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84961230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11164210 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149460246743360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38301942 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154519221804032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84961230 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46632920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156626998673920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170573625260032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170573649200128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚2743.741569259 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2106866886144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172778694874624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2098982994151 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154519203801088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46659288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 trainer/global_step‚1849 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154519221804032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11163597 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2106866886144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2099890969355 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84934860 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154519203801088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2107776869888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22941769 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172778935001600 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38301940 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154519221804032 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11163597 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2106866886144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2099890969355 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84934860 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154519203801088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2107776869888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22941769 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172778935001600 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38301940 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34105366 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84934860 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38301940 % _timestamp‚1.7885379484296093e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84961230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11164210 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149460246743360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38301942 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154519221804032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84961230 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46632920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156626998673920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170573625260032 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170573649200128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚2743.741569259 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2106866886144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚172778694874624 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2098982994151 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154519203801088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46659288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 trainer/global_step‚1849 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174994708718080 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2262943715840 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87169115 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47852991 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160680371857408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2161832850944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34983059 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87142771 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39289780 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47852991 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174994684777984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158518539006464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177257652433920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262727515648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185€Ü>ó'³íO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2162742821376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177257412293632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2153742975981 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 trainer/global_step‚1899 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47879333 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153332108876016 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160681299830784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240140288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216200192 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158518557009408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885380089825528e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47879333 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2161832850944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158518557009408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158518539006464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153332125915376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23532956 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23532954 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚155486776866353 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160680371857408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39289780 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2162742821376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34983059 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87142771 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39289780 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47852991 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174994684777984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158518539006464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177257652433920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262727515648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11450103 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11449476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39289782 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34982430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚155485851851997 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160681299830784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39289782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174994708718080 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2262943715840 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87169115 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47852991 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160680371857408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2161832850944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885380100135353e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158600176095232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚153 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160764040390144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158600158092288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2163864294912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47877822 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177349051069952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87187762 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2154860249259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39309942 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39309940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23545021 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87187762 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153411126470560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStmö[ÚatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158600158092288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2162954311168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2163864294912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47904190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175084910482432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23545019 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11455890 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160764040390144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155768224463 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47904190 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87214132 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39309940 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2264116647424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2162954311168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35000270 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177348796262912 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35000911 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87214132 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39309940 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2264116647424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2162954311168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35000270 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177348796262912 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35000911 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 % _timestamp‚1.7885380100135353e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158600176095232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚153 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160764040390144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158600158092288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2163864294912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47877822 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177349051069952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87187762 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2154860249259 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39309942 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39309940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23545021 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87187762 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153411126470560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158600158092288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2162954311168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2163864294912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47904190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175084910482432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23545019 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11455890 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160764040390144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0™WÊ<Ø2¹- _wandbruntime‚2805  _runtime‚2805bVhYºÓ 2loss/FBt4/assignment_loss‚0.017179964808747172 /loss/FRt3/detection_loss‚2.8663887977600098 epoch‚1 0loss/FBt2/assignment_loss‚0.2368733847644459 -loss/FRt1/detection_loss‚2.90241301444269 /loss/FBt1/detection_loss‚3.0165683892038135 2loss/SRqqt2/assignment_loss‚1.6797955948859453 1loss/SRqqt3/detection_loss‚2.0252952655156453 0loss/SRqqt4/detection_loss‚1.040326485641239 _runtime‚2805.327360763 _step‚154 0loss/FBt3/assignment_loss‚0.3811897507963103 1loss/SRqqt2/detection_loss‚3.0856003798544407 /loss/FBt3/detection_loss‚2.0221668105376396 /loss/FRt3/assignment_loss‚4.015620324760675 1loss/SRqqt3/assignment_loss‚1.960625760257244 /loss/FBt4/detection_loss‚3.2020005881786346 trainer/global_step‚1899 0loss/FRt4/assignment_loss‚2.3547414155073567 2loss/SRqqt1/assignment_loss‚1.7715974556548255 2loss/FBt1/assignment_loss‚0.010865797114092857 0loss/SRqqt1/detection_loss‚2.775761194527149 /loss/FBt2/detection_loss‚0.8328736243355606 %loss/total_loss‚49.56138026449603 % _timestamp‚1.7885380100165443e+09 /loss/FRt1/assignment_loss‚3.879332334764542 1loss/SRqqt4/assignment_loss‚1.463769585171268 .loss/FRt2/detection_loss‚2.933690195679665 .loss/FRt4/detection_loss‚1.723352538243028 0loss/FRt2/assignment_loss‚3.3633516120910643¼>T?^»Ø 2loss/FBt4/assignment_loss‚0.017179964808747172 /loss/FRt3/detection_loss‚2.8663887977600098 epoch‚1 0loss/FBt2/assignment_loss‚0.2368733847644459 -loss/FRt1/detection_loss‚2.90241301444269 /loss/FBt1/detection_loss‚3.0165683892038135 2loss/SRqqt2/assignment_loss‚1.6797955948859453 1loss/SRqqt3/detection_loss‚2.0252952655156453 0loss/SRqqt4/detection_loss‚1.040326485641239 _runtime‚2805.327360763 _step‚154 0loss/FBt3/assignment_loss‚0.3811897507963103 1loss/SRqqt2/detection_loss‚3.0856003798544407 /loss/FBt3/detection_loss‚2.0221668105376396 /loss/FRt3/assignment_loss‚4.015620324760675 1loss/SRqqt3/assignment_loss‚1.960625760257244 /loss/FBt4/detection_loss‚3.2020005881786346 trainer/global_step‚1899 0loss/FRt4/assignment_loss‚2.3547414155073567 2loss/SRqqt1/assignment_loss‚1.7715974556548255 2loss/FBt1/assignment_loss‚0.010865797114092857 0loss/SRqqt1/detection_loss‚2.775761194527149 /loss/FBt2/detection_loss‚0.8328736243355606 %loss/total_loss‚49.56138026449603 % _timestamp‚1.7885380100165443e+09 /loss/FRt1/assignment_loss‚3.879332334764542 1loss/SRqqt4/assignment_loss‚1.463769585171268 .loss/FRt2/detection_loss‚2.933690195679665 .loss/FRt4/detection_loss‚1.723352538243028 0loss/FRt2/assignment_loss‚3.3633516120910643šn’Iâž¼‚(:“ ¨ÙëÔ¾³ŸÑ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.097' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.019399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯î½j½‚(:ß ¨ÙëÔùÆßÓ) proc.memory.percent‚18.65999698638916 proc.cpu.threads‚17$ disk./.usageGB‚1178.443042755127 disk.sda2.in‚0 disk.sda3.out‚ 97.1171875 cpu‚25.349217105618237 network.recv‚927896 disk.dm-0.in‚ 6730.71484375! disk.dm-0.out‚54394.591796875 disk.sda2.out‚0 disk.drbd1260.out‚ 0.41015625& disk.nvme0c0n1.out‚54394.591796875$ disk.nvme0n1.out‚54394.591796875 disk.sda.in‚0 network.sent‚ 3.807091e+06) disk./.usagePercent‚79.08630137886887 disk.dm-1.out‚ 97.1171875 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda1.in‚0# memory_percent‚99.9624252319336# proc.memory.availableMB‚6.15625 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 6730.71484375! disk.nvme0n1.in‚ 6730.71484375 disk.sda.out‚ 97.1171875 disk.sda1.out‚0# proc.memory.rssMB‚ 3057.25390625l2 “¾‚(:ˆ ·ÙëÔÞ½ßÊ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á„ægs¿‚(:è ·ÙëÔ»÷öÍ disk.nvme0n1.in‚ 6814.2578125& disk.nvme0n1.out‚54543.27392578125 disk.sda.in‚0 disk.drbd1260.out‚ 0.41015625& proc.memory.availableMB‚ 5.57421875 cpu‚25.348114123501887) disk./.usagePercent‚79.08788017733252 disk.sda3.out‚ 97.54296875 disk.sda1.out‚0 disk.dm-0.in‚ 6814.2578125 disk.dm-1.in‚0 disk.dm-1.out‚ 97.54296875 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 6814.2578125 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0# memory_percent‚99.9659776687622# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 proc.cpu.threads‚17 network.sent‚ 3.905292e+06 network.recv‚933975$ disk./.usageGB‚1178.466567993164 disk.sda.out‚ 97.54296875# disk.dm-0.out‚54543.27392578125( disk.nvme0c0n1.out‚54543.27392578125¡¤‡òÀ‚(:’ ÆÙëÔ´ÞÌÍ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.098' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.81960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º‡~2hÁ‚(:Ý ÆÙëÔ£ÿ´Ô# disk.dm-0.out‚55024.71728515625 disk.drbd1260.out‚ 0.4140625 disk.nvme0c0n1.in‚6913.875) proc.memory.percent‚18.65999698638916 cpu‚25.346973970373497 proc.cpu.threads‚17% disk./.usageGB‚1178.4719352722168 disk.nvme0n1.in‚6913.875 disk.sda.in‚0 disk.sda.out‚ 97.95703125 disk.sda1.in‚0 disk.dm-1.out‚ 97.95703125 disk.sda2.out‚0# proc.memory.rssMB‚ 3057.25390625 network.recv‚937396 disk.sda3.out‚ 97.95703125 disk.sda1.out‚0 disk.sda2.in‚0& disk.nvme0n1.out‚55024.71728515625 disk.sda3.in‚0 disk.dm-0.in‚6913.875 disk.dm-1.in‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚55024.71728515625$ memory_percent‚99.96335506439209& proc.memory.availableMB‚ 6.00390625 network.sent‚ 3.910315e+06) disk./.usagePercent‚79.08824037993321Ï?ø³“‚(:ˆ ÕÙëÔ¸ýÕÌ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.776' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™R¼^Â(:Ó ÕÙëÔ²öÑ disk.sda1.out‚0 disk.dm-1.out‚ 98.6328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 98.6328125 disk.sda1.in‚0 disk.dm-0.in‚6913.875# disk.dm-0.out‚55311.96728515625 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.96490478515625# proc.memory.rssMB‚ 3057.25390625 network.sent‚ 3.915447e+06 network.recv‚941065 disk.sda.out‚ 98.6328125) proc.memory.percent‚18.65999698638916( disk.nvme0c0n1.out‚55311.96728515625 disk.nvme0n1.in‚6913.875 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd1260.out‚ 0.4140625 proc.memory.availableMB‚5.75 cpu‚25.34583109474538 proc.cpu.threads‚17) disk./.usagePercent‚79.08823781985787% disk./.usageGB‚1178.4718971252441 disk.nvme0c0n1.in‚6913.875& disk.nvme0n1.out‚55311.96728515625d…Ê{2Ä- _wandbruntime‚2865  _runtime‚2865'X?ÆÅÀ & lr-AdamW/pg1‚0.0016663307432809604 & lr-AdamW/pg2‚0.0016663307432809604 trainer/global_step‚1949 $ _timestamp‚1.788538070469342e+09 _runtime‚2865.780419117 _step‚155vI¶wËÆÅ trainer/global_step‚1949 $ _timestamp‚1.788538070469342e+09 _runtime‚2865.780419117 _step‚155 & lr-AdamW/pg1‚0.0016663307432809604 & lr-AdamW/pg2‚0.0016663307432809604›òS12Ç- _wandbruntime‚2865  _runtime‚2865àDÊ%ÈèO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35877513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89422017 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40297780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181827646510080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179505993940480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2217920275968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2321652569600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚1949 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89395673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162599511300608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885380704708676e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40297782 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24136204 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218830246400 _runtime‚2865.782147485 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159492608834305 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164817413573632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164818341547008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231751680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49124235 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162599511300608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35876911 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157282988603216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40297780 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162599493297664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157283005642576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40297782 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatÕCü›$sMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944Y jó'ÉíO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35877513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89422017 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40297780 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181827646510080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179505993940480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2217920275968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2321652569600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚1949 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89395673 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162599511300608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 % _timestamp‚1.7885380704708676e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40297782 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24136204 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218830246400 _runtime‚2865.782147485 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159492608834305 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164817413573632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164818341547008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231751680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49124235 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162599511300608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35876911 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157282988603216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40297780 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162599493297664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157283005642576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40297782 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219951719936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40317942 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157362006197760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162681112383488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2219041736192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49149092 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40317942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49149092 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35894796 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181918817713664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24148271 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11746527 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40317940 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35895399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179596195704832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49122724 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40317940 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89467034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179596219644928 % _timestamp‚1.7885380718102407e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159572743702127 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164901082106368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181919066229248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89467034 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162681130386432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162681112383488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2211645479571 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49122724 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚2867.122679228 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2219041736192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚603 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ñ¿ß>ý&Ì÷M FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219951719936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40317942 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157362006197760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162681112383488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2219041736192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49149092 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40317942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49149092 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35894796 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181918817713664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24148271 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11746527 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40317940 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35895399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179596195704832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224575488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49122724 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40317940 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89467034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179596219644928 % _timestamp‚1.7885380718102407e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159572743702127 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164901082106368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚181919066229248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89467034 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162681130386432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162681112383488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2211645479571 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49122724 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 _runtime‚2867.122679228 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2219041736192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚603 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 trainer/global_step‚1949 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184017279162880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91648575 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2274007700992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41305782 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168954455289856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168954455289856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186397435949568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50342795 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266405461193 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50342795 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2380405315584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24739456 % _timestamp‚1.7885381310589943e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168955383263232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2274007700992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265497486197 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41305782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12031714 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166680465591808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186397684478464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184017255222784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24739454 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168955383263232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166680465591808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163499365816613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185‚ Ìâó'ÝíO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41305782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12031714 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚166680465591808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186397684478464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184017255222784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24739454 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168955383263232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166680465591808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163499365816613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2926.370024702 _step‚160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2274917671424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36771786 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161233885369776 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41305780 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2274917671424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2380180726784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163500290830969 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚161233868330416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50369137 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184017279162880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91648575 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2274007700992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41305782 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168954455289856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168954455289856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186397435949568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50342795 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266405461193 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50342795 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2380405315584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24739456 % _timestamp‚1.7885381310589943e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚168955383263232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2274007700992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265497486197 ˆ´ìÃ2Þ- _wandbruntime‚2927  _runtime‚2927Û­#ë ßñM _runtime‚2927.363559225 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693566 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2275129161216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2276039144960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788538132051534e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41325942 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36789578 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163579500684435 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169038123822592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91719936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50393994 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166762066674688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41325940 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163580425698999 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2266614759475 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166762084677632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50367626 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267522734679 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚161 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91719936 OADeviceStatsMonitor.on_train_batch_end/requestedOùÅÚë_bytes.all.current‚ 925014564 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36788974 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24751521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166762084677632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12037455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161312885924960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12038057 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186489085937664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169037195835904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2381581070336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2381362786304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640`.$ü&àöM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163579500684435 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169038123822592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91719936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50393994 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166762066674688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41325940 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163580425698999 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2266614759475 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166762084677632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50367626 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267522734679 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚161 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91719936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36788974 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24751521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166762084677632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12037455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161312885924960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12038057 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186489085937664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169037195835904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2381581070336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2381362786304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚2927.363559225 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693566 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2275129161216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2276039144960 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788538132051534e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41325942 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36789578 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¡«zÐ 2á- _wandbruntime‚2927  _runtime‚2927Ö_äZâÔ /loss/FRt3/assignment_loss‚4.744884412735701 0loss/FBt4/assignment_loss‚0.3475152254104614 /loss/FRt1/detection_loss‚2.5560802497514863 /loss/FRt4/detection_loss‚1.6537199096656554 /loss/FBt2/detection_loss‚0.9431615532312083 trainer/global_step‚1999 2loss/SRqqt1/assignment_loss‚1.5837164631570286 2loss/SRqqt4/assignment_loss‚1.3685668692053334 # _timestamp‚1.78853813205471e+09 0loss/FRt4/assignment_loss‚2.4470819609780468 .loss/FBt4/detection_loss‚3.055357919798957 epoch‚1 1loss/FBt2/assignment_loss‚0.19837243767049223 1loss/FBt3/assignment_loss‚0.46843701100308976 0loss/SRqqt3/detection_loss‚2.385963240335154 0loss/SRqqt4/detection_loss‚1.048791747752244 _step‚162 0loss/FRt1/assignment_loss‚4.1183214042244884 _runtime‚2927.365559887 0loss/FRt2/assignment_loss‚2.6276086451960543 1loss/FBt1/assignment_loss‚0.26827130370828156 /loss/FRt3/detection_loss‚2.8349756821990013 1loss/SRqqt1/detection_loss‚2.2128000669884234 0loss/SRqqt2/detection_loss‚2.856491235891978 /loss/FBt1/detection_loss‚2.8201657042783848 .loss/FBt3/detection_loss‚2.445967996120453 %loss/total_loss‚48.90881734155987 2loss/SRqqt2/assignment_loss‚1.4647401407361031 2loss/SRqqt3/assignment_loss‚2.3684778047162434 /loss/FRt2/detection_loss‚2.0893483568055955 Ïç”_ãÙ 2loss/SRqqt2/assignment_loss‚1.4647401407361031 2loss/SRqqt3/assignment_loss‚2.3684778047162434 /loss/FRt2/detection_loss‚2.0893483568055955 /loss/FRt3/assignment_loss‚4.744884412735701 0loss/FBt4/assignment_loss‚0.3475152254104614 /loss/FRt1/detection_loss‚2.5560802497514863 /loss/FRt4/detection_loss‚1.6537199096656554 /loss/FBt2/detection_loss‚0.9431615532312083 trainer/global_step‚1999 2loss/SRqqt1/assignment_loss‚1.5837164631570286 2loss/SRqqt4/assignment_loss‚1.3685668692053334 # _timestamp‚1.78853813205471e+09 0loss/FRt4/assignment_loss‚2.4470819609780468 .loss/FBt4/detection_loss‚3.055357919798957 epoch‚1 1loss/FBt2/assignment_loss‚0.19837243767049223 1loss/FBt3/assignment_loss‚0.46843701100308976 0loss/SRqqt3/detection_loss‚2.385963240335154 0loss/SRqqt4/detection_loss‚1.048791747752244 _step‚162 0loss/FRt1/assignment_loss‚4.1183214042244884 _runtime‚2927.365559887 0loss/FRt2/assignment_loss‚2.6276086451960543 1loss/FBt1/assignment_loss‚0.26827130370828156 /loss/FRt3/detection_loss‚2.8349756821990013 1loss/SRqqt1/detection_loss‚2.2128000669884234 0loss/SRqqt2/detection_loss‚2.856491235891978 /loss/FBt1/detection_loss‚2.8201657042783848 .loss/FBt3/detection_loss‚2.445967996120453 %loss/total_loss‚48.90881734155987¢ z2é“ä‚(:ˆ  ÚëÔð©ÚÎ gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.651' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ]¬¨oå‚(:ä  ÚëÔüÒ disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.4296875 disk.dm-1.out‚ 101.46484375 disk.drbd1260.in‚0$ memory_percent‚99.96716976165771# proc.memory.rssMB‚ 3057.25390625 proc.cpu.threads‚17( disk./.usagePercent‚79.1050728753095 disk.nvme0n1.in‚ 7027.2421875 disk.dm-0.in‚ 7027.2421875! disk.dm-0.out‚56652.322265625& proc.memory.availableMB‚ 5.37890625) proc.memory.percent‚18.65999698638916 cpu‚25.34032740797501 network.sent‚ 4.046163e+06 network.recv‚963140% disk./.usageGB‚1178.7227516174316" disk.nvme0c0n1.in‚ 7027.2421875& disk.nvme0c0n1.out‚56652.322265625$ disk.nvme0n1.out‚56652.322265625 disk.sda3.in‚0 disk.sda3.out‚ 101.46484375 disk.sda.in‚0 disk.sda.out‚ 101.46484375 disk.sda1.in‚0GmhF’æ‚(:‡ ¯ÚëÔÎ߇Ì gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AÚìÛoç‚(:ä ¯ÚëÔ†¾ÊÑ network.recv‚969881" disk.nvme0c0n1.in‚ 7027.2421875 disk.sda1.in‚0) proc.memory.percent‚18.65999698638916 disk.sda3.out‚ 102.07421875 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 7027.2421875 disk.sda3.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 5.62890625 cpu‚25.33930279436337 network.sent‚ 4.157187e+06% disk./.usageGB‚1178.9737701416016$ disk.nvme0n1.out‚56997.951171875 disk.sda.in‚0! disk.dm-0.out‚56997.951171875 disk.dm-1.out‚ 102.07421875# proc.memory.rssMB‚ 3057.25390625 proc.cpu.threads‚17( disk./.usagePercent‚79.1219189390851 disk.sda2.in‚0 disk.drbd1260.out‚ 0.4296875& disk.nvme0c0n1.out‚56997.951171875 disk.sda.out‚ 102.07421875 disk.dm-0.in‚ 7027.2421875 disk.dm-1.in‚0$ memory_percent‚99.96564388275146osƒë“è‚(:ˆ ¾ÚëÔ’ëåË gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8778 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾æ,†té‚(:é ¾ÚëÔõ«ÖÎ network.sent‚ 4.160349e+06' disk.nvme0c0n1.out‚57297.4228515625! disk.nvme0n1.in‚ 7140.07421875 disk.sda.in‚0% proc.memory.availableMB‚ 5.6171875# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916$ disk./.usageGB‚1178.972770690918% disk.nvme0n1.out‚57297.5478515625 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚99.9657154083252 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 102.70703125" disk.dm-0.out‚57271.2978515625 disk.dm-1.in‚0 disk.dm-1.out‚ 102.70703125 cpu‚25.33832939329793 network.recv‚973353) disk./.usagePercent‚79.12185186511115 disk.sda2.out‚0 disk.drbd1260.out‚ 0.43359375# disk.nvme0c0n1.in‚ 7140.07421875 disk.sda.out‚ 102.70703125 disk.dm-0.in‚ 7140.07421875 proc.cpu.threads‚17Oµàžê‚(:“ ÍÚëÔÚ›˜Ì gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.944' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.388799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êþöUqë‚(:æ ÍÚëÔäƒßÎ% disk.nvme0n1.out‚57362.5595703125 disk.sda3.in‚0 disk.sda3.out‚ 103.19921875 disk.sda1.out‚0" disk.dm-0.out‚57362.5595703125$ memory_percent‚99.96566772460938 disk.sda.out‚ 103.19921875 disk.dm-0.in‚ 7140.07421875 disk.dm-1.out‚ 103.19921875! proc.memory.availableMB‚5.625# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 cpu‚25.33723205368364 network.sent‚ 4.16564e+06) disk./.usagePercent‚79.12185058507347 disk.drbd1260.out‚ 0.43359375# disk.nvme0c0n1.in‚ 7140.07421875! disk.nvme0n1.in‚ 7140.07421875 disk.sda.in‚0% disk./.usageGB‚1178.9727516174316 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚57362.5595703125 disk.sda1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚977053 disk.drbd1260.in‚0e£2ì- _wandbruntime‚2988  _runtime‚2988x¯›ªÇíÁ & lr-AdamW/pg1‚0.0016608290202079251 & lr-AdamW/pg2‚0.0016608290202079251 trainer/global_step‚2049 % _timestamp‚1.7885381934619274e+09 _runtime‚2988.773128402 _step‚163Fü ÌîÆ _runtime‚2988.773128402 _step‚163 & lr-AdamW/pg1‚0.0016608290202079251 & lr-AdamW/pg2‚0.0016608290202079251 trainer/global_step‚2049 % _timestamp‚1.7885381934619274e+09£ltüé2ï- _wandbruntime‚2988  _runtime‚2988AxXm &ðèO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170761419883008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚190967880861184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173091497006080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167507047813277 trainer/global_step‚2049 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42313782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12322383 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42313780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190967644915200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25342706 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2331005096448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51587697 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165184765096976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93901477 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51587697 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173092424979456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25342704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2330095126016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173091497006080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚164 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188528564385280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51614039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42313780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚2988.774683925 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331005096448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167506122798921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37665087 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51614039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170761401880064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173092424979456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235945984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2322282716301 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165184748057616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93927821 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42313782 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885381934635258e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170761401880064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚188528540445184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439104470016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173091497006080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167507047813277 trainer/global_step‚2049 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42313782 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12322383 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42313780 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190967644915200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25342706 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2331005096448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51587697 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165184765096976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93901477 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51587697 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173092424979456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25342704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2330095126016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173091497006080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚164 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188528564385280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51614039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42313780 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚2988.774683925 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2331005096448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167506122798921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37665087 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51614039 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170761401880064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚173092424979456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235945984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2322282716301 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165184748057616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93927821 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42313782 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885381934635258e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170761401880064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚188528540445184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2439104470016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42333940 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25354771 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191059030201344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188618790089728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167586257666743 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165263782691520 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37682967 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191059287105536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93972838 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51612528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2440264051712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12328845 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232964096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170843038968832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173174237552128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2331216586240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188618766149632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93972838 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173175165538816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170843020965888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170843038968832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42333940 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2322492014583 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51638896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚165 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51612528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42333942 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2323399989787 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332126569984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2331216586240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256904192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42333942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51638896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37683616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37682967 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191059287105536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93972838 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51612528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2440264051712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12328845 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232964096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170843038968832 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173174237552128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2331216586240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188618766149632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93972838 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173175165538816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170843020965888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170843038968832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42333940 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2322492014583 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51638896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _step‚165 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51612528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42333942 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2323399989787 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2332126569984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2331216586240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256904192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42333942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51638896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37683616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42333940 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25354771 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚647 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191059030201344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚188618790089728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167586257666743 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165263782691520 Evwù‚(:ì ÜÚëÔŽßèÐ# proc.memory.rssMB‚ 3057.25390625 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda3.in‚0) proc.memory.percent‚18.65999698638916 cpu‚25.336275071108332 proc.cpu.threads‚17) disk./.usagePercent‚79.12184674496045% disk./.usageGB‚1178.9726943969727 disk.dm-0.in‚ 7140.07421875 disk.dm-1.out‚ 103.55078125( disk.nvme0c0n1.out‚57732.62548828125! disk.nvme0n1.in‚ 7140.07421875& disk.nvme0n1.out‚57732.62548828125 disk.sda1.in‚0 network.recv‚981279# disk.dm-0.out‚57732.62548828125 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 103.55078125 disk.sda1.out‚0$ memory_percent‚99.96922016143799& proc.memory.availableMB‚ 5.04296875 network.sent‚ 4.171423e+06 disk.drbd1260.out‚0.4375# disk.nvme0c0n1.in‚ 7140.07421875 disk.sda3.out‚ 103.55078125 disk.sda.in‚0Pá8Þžú‚(:“ ëÚëÔ•×†Ì gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.699' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.339800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þú@¬vû‚(:ë ëÚëÔ×ljÐ proc.cpu.threads‚17 network.sent‚ 4.28018e+06 network.recv‚987654 disk.dm-0.in‚ 7140.07421875 disk.dm-1.in‚0& disk.nvme0n1.out‚58244.12548828125 disk.sda3.out‚ 103.92578125 disk.sda.out‚ 103.92578125$ memory_percent‚99.96612071990967% disk./.usageGB‚1178.9727516174316 disk.dm-1.out‚ 103.92578125 disk.drbd1260.in‚0 cpu‚25.335131487171466) disk./.usagePercent‚79.12185058507347 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚0.4375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚58244.12548828125# disk.nvme0c0n1.in‚ 7140.07421875( disk.nvme0c0n1.out‚58244.12548828125! disk.nvme0n1.in‚ 7140.07421875& proc.memory.availableMB‚ 5.55078125# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916â<’ü‚(:‡ úÚëÔåÍ¡Ë gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TØ÷wý‚(:ì úÚëÔ÷éþÍ% disk./.usageGB‚1179.4747886657715 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 7274.80078125 disk.dm-1.in‚0 disk.dm-1.out‚ 104.36328125 disk.sda3.out‚ 104.36328125 disk.sda.in‚0" disk.dm-0.out‚58575.7548828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4453125# disk.nvme0c0n1.in‚ 7274.80078125 network.recv‚990878 disk.sda3.in‚0' disk.nvme0c0n1.out‚58575.7548828125% disk.nvme0n1.out‚58575.7548828125$ memory_percent‚99.96764659881592& proc.memory.availableMB‚ 5.30078125 proc.cpu.threads‚17 disk.sda.out‚ 104.36328125! disk.nvme0n1.in‚ 7274.80078125# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 cpu‚25.334264170235993 network.sent‚ 4.287059e+06) disk./.usagePercent‚79.15554271262471„*Þz“þ‚(:ˆ ‰ÛëÔšæâÊ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚159.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸‡ž¼qÿ‚(:æ ‰ÛëÔÞŠðÎ disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.33319440387855) disk./.usagePercent‚79.15540600460142 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 104.7890625" disk.dm-0.out‚58786.7314453125$ memory_percent‚99.96767044067383# proc.memory.rssMB‚ 3057.25390625 disk.sda2.in‚0! disk.nvme0n1.in‚ 7274.80078125% disk.nvme0n1.out‚58786.7314453125 disk.dm-0.in‚ 7274.80078125 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚994455' disk.nvme0c0n1.out‚58786.7314453125 disk.sda3.in‚0 disk.sda3.out‚ 104.7890625 disk.dm-1.out‚ 104.7890625$ proc.memory.availableMB‚5.296875) proc.memory.percent‚18.65999698638916 network.sent‚ 4.290459e+06% disk./.usageGB‚1179.4727516174316 disk.sda2.out‚0 disk.drbd1260.out‚ 0.4453125# disk.nvme0c0n1.in‚ 7274.80078125i È”2€- _wandbruntime‚3051  _runtime‚3051¾)êLľ trainer/global_step‚2099 $ _timestamp‚1.788538255965351e+09 _runtime‚ 3051.2763895 _step‚167 & lr-AdamW/pg1‚0.0016578221323163944 & lr-AdamW/pg2‚0.0016578221323163944z¾¡&É‚Ã & lr-AdamW/pg1‚0.0016578221323163944 & lr-AdamW/pg2‚0.0016578221323163944 trainer/global_step‚2099 $ _timestamp‚1.788538255965351e+09 _runtime‚ 3051.2763895 _step‚167§×QÞ2ƒ- _wandbruntime‚3051  _runtime‚3051¿.¿î'„èO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885382559667542e+09 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171513804795585 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52832599 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174842374174208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38560332 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377251996413 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193039849607680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43321782 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195537672630272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43321780 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25945954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12613759 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43321782 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177228538722304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174842356171264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25945956 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2378159971409 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177229466695680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12614376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169135644824176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171512879781229 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52832599 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193039825667584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174842374174208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174842356171264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38559713 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2386182551040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2387092521472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195537921159168 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52858941 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177228538722304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2386182551040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169135627784816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96180723 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚3051.278019129 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2387092521472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2497846962688΢äó'…íO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174842374174208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174842356171264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38559713 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2386182551040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2387092521472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195537921159168 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52858941 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177228538722304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2386182551040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169135627784816 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96180723 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _runtime‚3051.278019129 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2387092521472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2497846962688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885382559667542e+09 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171513804795585 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52832599 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174842374174208 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38560332 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377251996413 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚193039849607680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43321782 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195537672630272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43321780 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25945954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12613759 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43321782 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177228538722304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174842356171264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25945956 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2378159971409 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177229466695680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12614376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169135644824176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171512879781229 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52832599 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193039825667584¨ñRZ72†- _wandbruntime‚3052  _runtime‚3052G ÊчòM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52857430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195629048613376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2387304011264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96225740 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171593014649051 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177311279268352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96199370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7885382570855973e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193130075312128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2378369269691 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174923975257088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12619518 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2498997241344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177312207255040 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177312207255040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38577537 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚3052.397589951 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 B8DeviceStatsMong‡rÿ'itor.on_train_batch_end/active.all.current‚26370 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43341940 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96199370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43341942 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12620145 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195629313906176½Òlý&ˆ÷M VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12620145 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195629313906176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52857430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195629048613376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2387304011264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96225740 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171593014649051 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177311279268352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96199370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7885382570855973e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193130075312128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 241352704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2378369269691 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174923975257088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12619518 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2498997241344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177312207255040 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177312207255040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38577537 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚3052.397589951 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43341940 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96199370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43341942 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344©þ׌2‰- _wandbruntime‚3052  _runtime‚3052Ì œ’WŠÑ /loss/FRt1/assignment_loss‚3.799832330809699 .loss/FRt4/detection_loss‚1.954673029951608 /loss/FBt2/detection_loss‚1.0057628683862276 .loss/FBt4/detection_loss‚4.390662391980489 /loss/FRt2/detection_loss‚2.7206419030061135 2loss/SRqqt1/assignment_loss‚1.8449939745995734 /loss/FRt3/detection_loss‚2.8757887376206264 0loss/SRqqt3/detection_loss‚2.876677668612936 /loss/FBt3/detection_loss‚2.0487557318475513 0loss/FRt2/assignment_loss‚2.5919972979105435 1loss/SRqqt2/assignment_loss‚2.292867712676525 0loss/FBt3/assignment_loss‚0.4104596473385013 0loss/SRqqt3/assignment_loss‚2.54082230510919 0loss/FBt2/assignment_loss‚0.1833834074305648 %loss/total_loss‚50.76596226703615 trainer/global_step‚2099 0loss/SRqqt4/detection_loss‚1.107720774583152 epoch‚1 2loss/FBt4/assignment_loss‚0.015789928069959085 0loss/FRt3/assignment_loss‚4.0470496607678275 /loss/FRt4/assignment_loss‚2.064133423108321 2loss/SRqqt4/assignment_loss‚1.3398665929304772 1loss/FBt1/assignment_loss‚0.25211613219562423 0loss/SRqqt1/detection_loss‚2.404365421997176 /loss/FRt1/detection_loss‚2.6660059889157615 1loss/SRqqt2/detection_loss‚2.2940854823047463 _runtime‚3052.400368945 _step‚170 /loss/FBt1/detection_loss‚3.0375098548829556 $ _timestamp‚1.788538257089355e+09y'þ\‹Ö 0loss/FRt2/assignment_loss‚2.5919972979105435 1loss/SRqqt2/assignment_loss‚2.292867712676525 0loss/FBt3/assignment_loss‚0.4104596473385013 0loss/SRqqt3/assignment_loss‚2.54082230510919 0loss/FBt2/assignment_loss‚0.1833834074305648 %loss/total_loss‚50.76596226703615 trainer/global_step‚2099 0loss/SRqqt4/detection_loss‚1.107720774583152 epoch‚1 2loss/FBt4/assignment_loss‚0.015789928069959085 0loss/FRt3/assignment_loss‚4.0470496607678275 /loss/FRt4/assignment_loss‚2.064133423108321 2loss/SRqqt4/assignment_loss‚1.3398665929304772 1loss/FBt1/assignment_loss‚0.25211613219562423 0loss/SRqqt1/detection_loss‚2.404365421997176 /loss/FRt1/detection_loss‚2.6660059889157615 1loss/SRqqt2/detection_loss‚2.2940854823047463 _runtime‚3052.400368945 _step‚170 /loss/FBt1/detection_loss‚3.0375098548829556 $ _timestamp‚1.788538257089355e+09 /loss/FRt1/assignment_loss‚3.799832330809699 .loss/FRt4/detection_loss‚1.954673029951608 /loss/FBt2/detection_loss‚1.0057628683862276 .loss/FBt4/detection_loss‚4.390662391980489 /loss/FRt2/detection_loss‚2.7206419030061135 2loss/SRqqt1/assignment_loss‚1.8449939745995734 /loss/FRt3/detection_loss‚2.8757887376206264 0loss/SRqqt3/detection_loss‚2.876677668612936 /loss/FBt3/detection_loss‚2.0487557318475513ªð >žŒ‚(:“ ˜ÛëÔÚÖÎ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.167399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ä$l‚(:á ˜ÛëÔöŒÂÓ' disk.nvme0c0n1.out‚58988.4423828125 disk.sda3.out‚ 105.3203125 disk.dm-1.in‚0# proc.memory.rssMB‚ 3057.25390625 network.sent‚ 4.296203e+06$ disk./.usageGB‚1179.472713470459% disk.nvme0n1.out‚58988.4423828125 disk.sda1.in‚0 disk.dm-0.in‚ 7274.80078125 network.recv‚998562# disk.nvme0c0n1.in‚ 7274.80078125! disk.nvme0n1.in‚ 7274.80078125 disk.sda3.in‚0 disk.sda.out‚ 105.3203125" disk.dm-0.out‚58988.4423828125# memory_percent‚99.9664306640625 proc.memory.availableMB‚5.5) disk./.usagePercent‚79.15540344452607 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 105.3203125) proc.memory.percent‚18.65999698638916 cpu‚25.332150312424698 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.44921875õT‘Ž‚(:† §ÛëÔÄ—»Ì gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 á2F}‚(:ò §ÛëÔÜÇžÑ disk.sda1.out‚0% proc.memory.availableMB‚ 5.5078125# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 cpu‚25.331315126129258) disk./.usagePercent‚79.15540600460142% disk.nvme0n1.out‚59362.2783203125 disk.sda.out‚ 105.80859375 disk.sda1.in‚0" disk.dm-0.out‚59362.2783203125 disk.dm-1.out‚ 105.80859375 proc.cpu.threads‚17% disk./.usageGB‚1179.4727516174316 disk.sda2.out‚0 disk.drbd1260.out‚ 0.44921875 disk.sda3.in‚0 disk.dm-0.in‚ 7274.80078125$ memory_percent‚99.96638298034668 network.recv‚ 1.004576e+06 disk.sda2.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 7274.80078125' disk.nvme0c0n1.out‚59362.2783203125! disk.nvme0n1.in‚ 7274.80078125 disk.sda.in‚0 disk.dm-1.in‚0 network.sent‚ 4.404899e+06 disk.sda3.out‚ 105.80859375­–Áp“‚(:ˆ ¶ÛëÔü‡ÂÌ gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©ž†¬|‘‚(:ñ ¶ÛëÔ¾—ŽÍ disk.dm-0.in‚ 7355.07421875$ memory_percent‚99.96488094329834) proc.memory.percent‚18.65999698638916 cpu‚25.33019874758351 disk.sda3.in‚0 network.sent‚ 4.411236e+06 network.recv‚ 1.008118e+06 disk.sda2.out‚0 disk.drbd1260.out‚0.453125! disk.nvme0n1.in‚ 7355.07421875 disk.sda.in‚0 disk.dm-1.out‚ 106.25390625 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 7355.07421875 disk.sda1.out‚0& proc.memory.availableMB‚ 5.75390625# proc.memory.rssMB‚ 3057.25390625 proc.cpu.threads‚17 disk.sda3.out‚ 106.25390625( disk./.usagePercent‚79.1554072846391$ disk./.usageGB‚1179.472770690918( disk.nvme0c0n1.out‚59617.56005859375 disk.sda.out‚ 106.25390625 disk.sda1.in‚0# disk.dm-0.out‚59617.56005859375 disk.dm-1.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚59617.56005859375¨íÕ’’‚(:‡ ÅÛëÔä’Ì gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00Þà–^“‚(:Ó ÅÛëÔØ«áÐ disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚59963.4326171875$ memory_percent‚99.96640682220459) disk./.usagePercent‚79.15540600460142% disk./.usageGB‚1179.4727516174316 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.453125 disk.nvme0c0n1.in‚7426.125) proc.memory.percent‚18.65999698638916" disk.dm-0.out‚59963.4326171875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚59963.4326171875 disk.nvme0n1.in‚7426.125& proc.memory.availableMB‚ 5.50390625 cpu‚25.329360300173942 proc.cpu.threads‚17 network.sent‚ 4.414422e+06 disk.sda3.out‚106.625 disk.dm-0.in‚7426.125 disk.dm-1.out‚106.625 disk.sda2.in‚0# proc.memory.rssMB‚ 3057.25390625 network.recv‚ 1.011441e+06 disk.sda.in‚0 disk.sda.out‚106.625Óû<2”- _wandbruntime‚3111  _runtime‚3111Kߙĕ¾ trainer/global_step‚2149 $ _timestamp‚1.788538316136616e+09 _runtime‚3111.447515623 _step‚171 % lr-AdamW/pg1‚0.001654645383853002 % lr-AdamW/pg2‚0.001654645383853002CÁïéÉ–Ã % lr-AdamW/pg1‚0.001654645383853002 % lr-AdamW/pg2‚0.001654645383853002 trainer/global_step‚2149 $ _timestamp‚1.788538316136616e+09 _runtime‚3111.447515623 _step‚171«§rbv2—- _wandbruntime‚3111  _runtime‚3111{q'˜çO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2556517036032 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175520561777893 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26549204 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚2149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181366508411904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚172 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12905642 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197551110889984 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175519636763537 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173086524551376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181365580438528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39454217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98433625 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178923328465408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26549206 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44329780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54077501 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173086507512016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442269976064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200107865969152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181366508411904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44329780 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44329782 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442269976064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181365580438528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200107627926016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54103843 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2434037226517 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2443179946496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚3111.448620235 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44329782 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197551134830080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54103843 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178923310462464 XJDeviceStatsMonitor.on_train_batch_start/requestedÐ`V¼|_bytes.small_pool.current‚ 907974996 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640È— ò'™ìO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2443179946496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚3111.448620235 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44329782 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197551134830080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54103843 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178923310462464 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2556517036032 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175520561777893 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26549204 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚2149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181366508411904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238043136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _step‚172 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12905642 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197551110889984 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175519636763537 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173086524551376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181365580438528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39454217 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98433625 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178923328465408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26549206 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44329780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54077501 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173086507512016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442269976064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200107865969152 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181366508411904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44329780 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44329782 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442269976064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181365580438528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200107627926016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54103843 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2434037226517¬UšŽ>2š- _wandbruntime‚3112  _runtime‚3112oÕ åø&›òM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181448320984576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200199048596480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44349942 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179004947551232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26561271 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2443391436288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2557712002048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54102332 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2557923994624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179004929548288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179004947551232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175600696645923 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2444301420032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98452272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44349940 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98478642 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12911298 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2434246524799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26561269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 trainer/global_step‚2149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173165542145920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2435154500003 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54128700 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44349942 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173165525106560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39471958 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235932672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39472569 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181449248971264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚1800294403¢ý&œ÷M UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2443391436288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2557712002048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54102332 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2557923994624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179004929548288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179004947551232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175600696645923 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2444301420032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98452272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44349940 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98478642 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12911298 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2434246524799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26561269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 trainer/global_step‚2149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173165542145920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2435154500003 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54128700 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44349942 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173165525106560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39471958 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235932672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39472569 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181449248971264 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181448320984576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200199048596480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44349942 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179004947551232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26561271­®[ä$2- _wandbruntime‚3112  _runtime‚3112:[ŒëWžÑ epoch‚1 # _timestamp‚1.78853831714104e+09 0loss/FBt2/assignment_loss‚0.1193513870548891 0loss/FRt4/assignment_loss‚2.5718191963131143 .loss/FRt3/detection_loss‚2.914533886217302 /loss/FRt4/detection_loss‚1.7957822368115235 1loss/SRqqt3/detection_loss‚2.5516897755510666 /loss/FBt4/detection_loss‚3.2921112775802612 0loss/FBt3/assignment_loss‚0.2155875474460906 1loss/SRqqt1/detection_loss‚2.5775770161833083 %loss/total_loss‚47.91711656986503 trainer/global_step‚2149 2loss/FBt4/assignment_loss‚0.060864485986530784 /loss/FRt1/assignment_loss‚4.390378421003168 /loss/FRt2/assignment_loss‚2.436550675606241 /loss/FRt3/assignment_loss‚4.414458267150387 2loss/SRqqt1/assignment_loss‚1.9459058429513658 2loss/SRqqt2/assignment_loss‚1.5884614363312721 1loss/SRqqt3/assignment_loss‚2.232099621611483 1loss/SRqqt2/detection_loss‚2.0749737607936063 1loss/FBt1/assignment_loss‚0.18878881087402988 /loss/FBt2/detection_loss‚0.7281606689044465 _runtime‚3112.451815454 _step‚174 .loss/FRt1/detection_loss‚2.829621578256289 0loss/SRqqt4/assignment_loss‚1.42520706304349 1loss/SRqqt4/detection_loss‚1.1100897824958615 .loss/FBt1/detection_loss‚2.335230004042387 /loss/FRt2/detection_loss‚1.8763166288940274 /loss/FBt3/detection_loss‚2.2415571987628935Åú\7ªŸÖ 0loss/FBt2/assignment_loss‚0.1193513870548891 0loss/FRt4/assignment_loss‚2.5718191963131143 .loss/FRt3/detection_loss‚2.914533886217302 /loss/FRt4/detection_loss‚1.7957822368115235 1loss/SRqqt3/detection_loss‚2.5516897755510666 /loss/FBt4/detection_loss‚3.2921112775802612 0loss/FBt3/assignment_loss‚0.2155875474460906 1loss/SRqqt1/detection_loss‚2.5775770161833083 %loss/total_loss‚47.91711656986503 trainer/global_step‚2149 2loss/FBt4/assignment_loss‚0.060864485986530784 /loss/FRt1/assignment_loss‚4.390378421003168 /loss/FRt2/assignment_loss‚2.436550675606241 /loss/FRt3/assignment_loss‚4.414458267150387 2loss/SRqqt1/assignment_loss‚1.9459058429513658 2loss/SRqqt2/assignment_loss‚1.5884614363312721 1loss/SRqqt3/assignment_loss‚2.232099621611483 1loss/SRqqt2/detection_loss‚2.0749737607936063 1loss/FBt1/assignment_loss‚0.18878881087402988 /loss/‡ ¼™²FBt2/detection_loss‚0.7281606689044465 _runtime‚3112.451815454 _step‚174 .loss/FRt1/detection_loss‚2.829621578256289 0loss/SRqqt4/assignment_loss‚1.42520706304349 1loss/SRqqt4/detection_loss‚1.1100897824958615 .loss/FBt1/detection_loss‚2.335230004042387 /loss/FRt2/detection_loss‚1.8763166288940274 /loss/FBt3/detection_loss‚2.2415571987628935 epoch‚1 # _timestamp‚1.78853831714104e+09®-±]7“ ‚(:ˆ ÔÛëÔñ§“Í gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Ni/Z¡‚(:Ï ÔÛëÔ‘ŠàÏ& proc.memory.availableMB‚ 8.16796875) disk./.usagePercent‚79.12917828872604 disk.sda2.out‚0 disk.drbd1260.out‚ 0.45703125 disk.nvme0c0n1.in‚7426.125 disk.sda1.in‚0# proc.memory.rssMB‚ 3057.25390625 cpu‚25.328290936831927 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 107.09375 disk.dm-1.out‚ 107.09375) proc.memory.percent‚18.65999698638916 network.sent‚ 4.420419e+06 disk.drbd1260.in‚0# disk.nvme0c0n1.out‚ 60305.015625 disk.nvme0n1.in‚7426.125! disk.nvme0n1.out‚ 60305.015625 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚99.95014667510986 network.recv‚ 1.015718e+06% disk./.usageGB‚1179.0819396972656 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 107.09375 disk.dm-0.in‚7426.125 disk.dm-0.out‚ 60305.015625=@¢Ñ“¢‚(:ˆ ãÛëÔÀáÆÌ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îȃèe£‚(:Ú ãÛëÔê“Ð disk.sda1.in‚0 disk.dm-0.in‚7426.125 disk.dm-1.in‚0 disk.nvme0n1.in‚7426.125 disk.sda3.in‚0 disk.sda3.out‚107.6875 disk.sda.in‚0 disk.sda1.out‚0# disk.dm-0.out‚60578.75830078125# proc.memory.rssMB‚ 3057.25390625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚99.94714260101318& proc.memory.availableMB‚ 8.66015625) disk./.usagePercent‚79.12918084880137 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.45703125( disk.nvme0c0n1.out‚60578.75830078125 disk.sda.out‚107.6875 disk.dm-1.out‚107.6875) proc.memory.percent‚18.65999698638916 cpu‚25.327444963380792 network.sent‚ 4.529197e+06 network.recv‚ 1.02217e+06% disk./.usageGB‚1179.0819778442383 disk.nvme0c0n1.in‚7426.125& disk.nvme0n1.out‚60578.75830078125“‡“¤‚(:ˆ òÛëÔÅ“ñÊ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ç-n¥‚(:ã òÛëÔÕÎàÎ( disk.nvme0c0n1.out‚60790.43017578125& disk.nvme0n1.out‚60790.43017578125 disk.sda3.in‚0 disk.sda.out‚ 108.1015625 network.sent‚ 4.534243e+06 network.recv‚ 1.025871e+06) disk./.usagePercent‚79.12921361776576# disk.dm-0.out‚60790.43017578125 disk.nvme0c0n1.in‚7426.125 disk.nvme0n1.in‚7426.125 disk.sda3.out‚ 108.1015625$ memory_percent‚99.94714260101318 disk.sda1.out‚0 disk.dm-0.in‚7426.125 disk.dm-1.in‚0& proc.memory.availableMB‚ 8.66015625# proc.memory.rssMB‚ 3057.25390625 proc.cpu.threads‚17% disk./.usageGB‚1179.0824661254883 disk.dm-1.out‚ 108.1015625 disk.sda2.in‚0 disk.sda.in‚0) proc.memory.percent‚18.65999698638916 cpu‚25.326400225777395 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4609375G7“¦‚(:ˆ ÜëÔáþÆË gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.057' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D>4êx§‚(:í ÜëÔàÇ‚Ì disk.sda.in‚0" disk.dm-0.out‚61067.7060546875& proc.memory.availableMB‚ 8.65234375 disk.nvme0n1.in‚ 7541.2734375) disk./.usagePercent‚79.12784013734456 disk.sda2.out‚0 disk.sda.out‚ 108.41015625" memory_percent‚99.947190284729 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 7541.2734375# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 cpu‚25.325621623239567% disk.nvme0n1.out‚61067.7060546875 network.sent‚ 4.539425e+06 network.recv‚ 1.029708e+06 disk.dm-1.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 7541.2734375' disk.nvme0c0n1.out‚61067.7060546875 disk.sda3.in‚0% disk./.usageGB‚1179.0620002746582 disk.dm-1.out‚ 108.41015625 disk.sda3.out‚ 108.41015625Ä92¨- _wandbruntime‚3171  _runtime‚3171e¡ËÇ©Á _step‚175 & lr-AdamW/pg1‚0.0016512994368473666 & lr-AdamW/pg2‚0.0016512994368473666 trainer/global_step‚2199 % _timestamp‚1.7885383766244073e+09 _runtime‚3171.935622171õîÝÌªÆ & lr-AdamW/pg1‚0.0016512994368473666 & lr-AdamW/pg2‚0.0016512994368473666 trainer/global_step‚2199 % _timestamp‚1.7885383766244073e+09 _runtime‚3171.935622171 _step‚175¯°i U2«- _wandbruntime‚3171  _runtime‚3171@Ô^öò'¬ìO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2498357401088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179526393745845 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183004264753664 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55322403 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40349804 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202062396112384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚3171.937607232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185502622154752 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40349196 _step‚176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27152456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13197348 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185503550128128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45337782 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 trainer/global_step‚2199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55348745 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179527318760201 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177037387239216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45337780 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183004282756608 % _timestamp‚1.7885383766261168e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100660183 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45337780 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2615645393920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185503550128128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55348745 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177037404278576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27152454 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚183004282756608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13196742 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2615431290880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2499267371520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185502622154752 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185502622154752 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40349804 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202062396112384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚3171.937607232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185502622154752 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40349196 _step‚176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27152456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13197348 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185503550128128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45337782 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 trainer/global_step‚2199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55348745 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179527318760201 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚177037387239216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45337780 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183004282756608 % _timestamp‚1.7885383766261168e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100660183 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45337780 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2615645393920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185503550128128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55348745 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177037404278576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27152454 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚183004282756608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13196742 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214103040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2615431290880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2499267371520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096°ŒçË®2®- _wandbruntime‚3173  _runtime‚3173£â})Ô¯öM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2616600221696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183085883839488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2500388845056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179606528613667 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100705174 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55373602 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183085901842432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27164519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202152621816832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100731544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚3173.251700891 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177116404833760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2616826894336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100705174 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚183085901842432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183085883839488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185586290687488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40367074 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13202555 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570èg F'°ûM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2616600221696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183085883839488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2500388845056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179606528613667 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100705174 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55373602 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183085901842432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27164519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202152621816832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100731544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚3173.251700891 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177116404833760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2616826894336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100705174 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚183085901842432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183085883839488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185586290687488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40367074 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13202555 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570±—ù2±- _wandbruntime‚3173  _runtime‚3173ëŸX*V²Ð 0loss/FRt3/assignment_loss‚4.2625563530360955 /loss/FRt4/assignment_loss‚2.327918250876737 trainer/global_step‚2199 $ _timestamp‚1.788538377942994e+09 2loss/SRqqt2/assignment_loss‚2.1746988222002983 0loss/FBt4/assignment_loss‚0.5927618026422957 /loss/FRt3/detection_loss‚2.8414408739875343 &loss/total_loss‚51.768564415204054 0loss/FBt1/assignment_loss‚0.3367919886109626 0loss/FBt2/assignment_loss‚0.4037014998458957 .loss/FBt1/detection_loss‚2.895763966772291 .loss/FRt1/detection_loss‚2.572829687060454 .loss/FRt2/detection_loss‚2.681164968871709 2loss/SRqqt4/assignment_loss‚1.3211988823942276 /loss/FBt3/detection_loss‚2.3556382168423045 /loss/FRt4/detection_loss‚1.5933597263128283 0loss/SRqqt3/detection_loss‚2.424000411198057 _runtime‚3173.254268149 /loss/FRt1/assignment_loss‚3.605889454866067 /loss/FRt2/assignment_loss‚3.433259974826466 2loss/SRqqt1/assignment_loss‚1.7266506659984588 0loss/FBt3/assignment_loss‚0.3577879909753854 -loss/FBt4/detection_loss‚4.05450705687205 epoch‚1 1loss/SRqqt1/detection_loss‚2.9527389454841613 1loss/SRqqt2/detection_loss‚2.7482801352938018 1loss/SRqqt4/detection_loss‚1.0216885778207738 _step‚178 2loss/SRqqt3/assignment_loss‚1.9932029946097012 /loss/FBt2/detection_loss‚1.0907331678054986›“Ô[³Õ .loss/FRt2/detection_loss‚2.681164968871709 2loss/SRqqt4/assignment_loss‚1.3211988823942276 /loss/FBt3/detection_loss‚2.3556382168423045 /loss/FRt4/detection_loss‚1.5933597263128283 0loss/SRqqt3/detection_loss‚2.424000411198057 _runtime‚3173.254268149 /loss/FRt1/assignment_loss‚3.605889454866067 /loss/FRt2/assignment_loss‚3.433259974826466 2loss/SRqqt1/assignment_loss‚1.7266506659984588 0loss/FBt3/assignment_loss‚0.3577879909753854 -loss/FBt4/detection_loss‚4.05450705687205 epoch‚1 1loss/SRqqt1/detection_loss‚2.9527389454841613 1loss/SRqqt2/detection_loss‚2.7482801352938018 1loss/SRqqt4/detection_loss‚1.0216885778207738 _step‚178 2loss/SRqqt3/assignment_loss‚1.9932029946097012 /loss/FBt2/detection_loss‚1.0907331678054986 0loss/FRt3/assignment_loss‚4.2625563530360955 /loss/FRt4/assignment_loss‚2.327918250876737 trainer/global_step‚2199 $ _timestamp‚1.788538377942994e+09 2loss/SRqqt2/assignment_loss‚2.1746988222002983 0loss/FBt4/assignment_loss‚0.5927618026422957 /loss/FRt3/detection_loss‚2.8414408739875343 &loss/total_loss‚51.768564415204054 0loss/FBt1/assignment_loss‚0.3367919886109626 0loss/FBt2/assignment_loss‚0.4037014998458957 .loss/FBt1/detection_loss‚2.895763966772291 .loss/FRt1/detection_loss‚2.572829687060454²šœk(‘´‚(:† ÜëÔÀö¨Í gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚133.77' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.754 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:¼ÌUpµ‚(:å ÜëÔˆ€œÑ disk.sda.out‚108.9375 disk.sda1.in‚0% proc.memory.availableMB‚ 8.3671875# proc.memory.rssMB‚ 3057.25390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.46484375" disk.nvme0c0n1.in‚ 7541.2734375 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.out‚108.9375 proc.cpu.threads‚17 network.sent‚ 4.545202e+06% disk./.usageGB‚1179.0871124267578( disk.nvme0c0n1.out‚61413.74658203125 disk.dm-0.in‚ 7541.2734375# disk.dm-0.out‚61413.74658203125 disk.dm-1.in‚0$ memory_percent‚99.94893074035645) disk./.usagePercent‚79.12952543494247 disk.nvme0n1.in‚ 7541.2734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚18.65999698638916 cpu‚25.32472132334681 network.recv‚ 1.033946e+06& disk.nvme0n1.out‚61413.74658203125 disk.sda3.out‚108.9375»´¹ùž¶‚(:“ ŸÜëÔ˨ŒÌ gpu.0.gpu‚21.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.187' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.437400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´õ¤x·‚(:í ŸÜëÔëÎ network.sent‚ 4.654194e+06 network.recv‚ 1.040828e+06 disk.sda3.out‚ 109.3515625 disk.dm-0.in‚ 7541.2734375# proc.memory.rssMB‚ 3057.25390625 cpu‚25.32387446436999) disk./.usagePercent‚79.12952927505548 disk.sda1.out‚0$ memory_percent‚99.94745254516602$ proc.memory.availableMB‚8.609375) proc.memory.percent‚18.65999698638916 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 109.3515625 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 7541.2734375( disk.nvme0c0n1.out‚61793.01611328125 disk.nvme0n1.in‚ 7541.2734375 disk.sda3.in‚0# disk.dm-0.out‚61793.01611328125% disk./.usageGB‚1179.0871696472168 disk.drbd1260.out‚ 0.46484375& disk.nvme0n1.out‚61793.01611328125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 109.3515625 disk.sda2.in‚0 disk.sda2.out‚0€´7‰‘¸‚(:† ®ÜëÔܾùÌ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 þu¹‚(:ê ®ÜëÔÓö‹Ò' disk.nvme0c0n1.out‚62259.1630859375 disk.sda3.out‚ 109.85546875 disk.dm-1.out‚ 109.85546875$ proc.memory.availableMB‚8.359375# proc.memory.rssMB‚ 3057.25390625 proc.cpu.threads‚17 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 7541.2734375 disk.sda1.in‚0 disk.dm-0.in‚ 7541.2734375) proc.memory.percent‚18.65999698638916 cpu‚25.32294631399859 network.sent‚ 4.659147e+06% disk./.usageGB‚1179.0871887207031 disk.sda.out‚ 109.85546875 disk.sda1.out‚0" disk.dm-0.out‚62259.1630859375 disk.dm-1.in‚0 network.recv‚ 1.044335e+06 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 7541.2734375% disk.nvme0n1.out‚62259.1630859375 disk.sda3.in‚0 disk.sda.in‚0$ memory_percent‚99.94897842407227) disk./.usagePercent‚79.12953055509315 disk.drbd1260.out‚0.46875rªÌzžº‚(:“ ½ÜëÔ€ÁËÌ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©¶>vz»‚(:ï ½ÜëÔôœÿÐ" disk.dm-0.out‚62407.6474609375) proc.memory.percent‚18.65999698638916 cpu‚25.322017501989166 proc.cpu.threads‚17) disk./.usagePercent‚79.12952927505548% disk./.usageGB‚1179.0871696472168' disk.nvme0c0n1.out‚62407.6474609375 disk.sda3.out‚ 110.53515625 disk.sda1.in‚0 disk.dm-0.in‚ 7729.87890625 network.sent‚ 4.664294e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 110.53515625 disk.sda1.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 3057.25390625 network.recv‚ 1.047878e+06 disk.sda2.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 7729.87890625 disk.sda.in‚0 disk.dm-1.out‚ 110.53515625$ memory_percent‚99.94750022888184% proc.memory.availableMB‚ 8.6015625 disk.drbd1260.out‚0.46875! disk.nvme0n1.in‚ 7729.87890625% disk.nvme0n1.out‚62407.64746093750±¾2¼- _wandbruntime‚3233  _runtime‚3233/D,tǽÁ & lr-AdamW/pg1‚0.0016477849885898287 & lr-AdamW/pg2‚0.0016477849885898287 trainer/global_step‚2249 % _timestamp‚1.7885384385749476e+09 _runtime‚3233.885947202 _step‚179Ë…‘Ì¾Æ _step‚179 & lr-AdamW/pg1‚0.0016477849885898287 & lr-AdamW/pg2‚0.0016477849885898287 trainer/global_step‚2249 % _timestamp‚1.7885384385749476e+09 _runtime‚3233.885947202³DƇô2¿- _wandbruntime‚3233  _runtime‚3233¨*Ô»ò'ÀìO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27755704 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56567305 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2555354796544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209248056630272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2554444826112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189640591844352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2674351355392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7885384385762823e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189639663870976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189639663870976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183533150728153 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41244852 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13488519 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206573705274880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189640591844352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2249 _step‚180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13489146 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56593647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187085219044864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2555354796544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46345782 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41244223 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183534075742509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187085237047808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180988284005776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46345782 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102913085 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46345780 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206573681334784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209247803907072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102939429 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102939429 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2554444826112 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189640591844352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2674351355392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7885384385762823e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189639663870976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189639663870976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252723200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183533150728153 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41244852 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13488519 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206573705274880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189640591844352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2249 _step‚180 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13489146 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56593647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187085219044864 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2555354796544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46345782 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41244223 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183534075742509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187085237047808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228783104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180988284005776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46345782 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102913085 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46345780 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206573681334784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209247803907072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0´×Vš2Â- _wandbruntime‚3235  _runtime‚3235ÌÝlÀü&ÃöM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218284032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102958076 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189723332403712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2675522254336 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187166838130688 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187166838130688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56618504 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41262133 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27767769 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚181 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675303970304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13494364 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181067284560960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46365940 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102984446 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46365940 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2555566286336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2555566286336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚206663907039232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181067301600320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189722404417024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56618504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187166856133632 % _timestamp‚1.7885384397306056e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41262763 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209339453233664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚206663930979328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚2249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2556476270080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183614210610539 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209339211009536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56592136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46365942 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÄR€Ÿ'ÄûM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13494364 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181067284560960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46365940 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102984446 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46365940 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2555566286336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2555566286336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚206663907039232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181067301600320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189722404417024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56618504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242224128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187166856133632 % _timestamp‚1.7885384397306056e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41262763 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209339453233664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚206663930979328 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 trainer/global_step‚2249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2556476270080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183614210610539 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209339211009536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56592136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46365942 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218284032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102958076 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189723332403712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2675522254336 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187166838130688 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187166838130688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56618504 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41262133 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27767769 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚181 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675303970304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640µ,Ò<€2Å- _wandbruntime‚3235  _runtime‚3235îSG"YÆÓ 2loss/SRqqt3/assignment_loss‚2.1283580977469683 1loss/FBt2/assignment_loss‚0.18842022812755022 /loss/FRt1/detection_loss‚3.4132645778034045 .loss/FRt2/detection_loss‚2.284220148705774 1loss/SRqqt1/detection_loss‚2.4028055333509677 .loss/FBt1/detection_loss‚2.305411282926798 1loss/FBt3/assignment_loss‚0.19420376425768993 /loss/FRt4/detection_loss‚1.6874976674821882 /loss/FRt3/assignment_loss‚4.492519557476044 2loss/SRqqt4/assignment_loss‚1.1507733038180517 _runtime‚3235.044253634 _step‚182 &loss/total_loss‚50.351716688811656 .loss/FRt3/detection_loss‚3.124195377031962 0loss/SRqqt2/detection_loss‚2.014572901385171 /loss/FRt4/assignment_loss‚2.192105824487251 2loss/SRqqt2/assignment_loss‚2.2142468103340693 .loss/FBt3/detection_loss‚3.039744865894318 trainer/global_step‚2249 0loss/FRt2/assignment_loss‚2.9715670893589654 0loss/FBt1/assignment_loss‚0.3622712998285351 $ _timestamp‚1.788538439733377e+09 /loss/FRt1/assignment_loss‚4.348239307818205 1loss/SRqqt1/assignment_loss‚2.046156463099689 2loss/FBt4/assignment_loss‚0.001198788150213659 0loss/SRqqt3/detection_loss‚2.910940627257029 1loss/SRqqt4/detection_loss‚1.0606831798032552 epoch‚1 /loss/FBt2/detection_loss‚0.7548999115289456 /loss/FBt4/detection_loss‚3.0634200811386108åµXoÇØ /loss/FBt2/detection_loss‚0.7548999115289456 /loss/FBt4/detection_loss‚3.0634200811386108 2loss/SRqqt3/assignment_loss‚2.1283580977469683 1loss/FBt2/assignment_loss‚0.18842022812755022 /loss/FRt1/detection_loss‚3.4132645778034045 .loss/FRt2/detection_loss‚2.284220148705774 1loss/SRqqt1/detection_loss‚2.4028055333509677 .loss/FBt1/detection_loss‚2.305411282926798 1loss/FBt3/assignment_loss‚0.19420376425768993 /loss/FRt4/detection_loss‚1.6874976674821882 /loss/FRt3/assignment_loss‚4.492519557476044 2loss/SRqqt4/assignment_loss‚1.1507733038180517 _runtime‚3235.044253634 _step‚182 &loss/total_loss‚50.351716688811656 .loss/FRt3/detection_loss‚3.124195377031962 0loss/SRqqt2/detection_loss‚2.014572901385171 /loss/FRt4/assignment_loss‚2.192105824487251 2loss/SRqqt2/assignment_loss‚2.2142468103340693 .loss/FBt3/detection_loss‚3.039744865894318 trainer/global_step‚2249 0loss/FRt2/assignment_loss‚2.9715670893589654 0loss/FBt1/assignment_loss‚0.3622712998285351 $ _timestamp‚1.788538439733377e+09 /loss/FRt1/assignment_loss‚)_Vï4.348239307818205 1loss/SRqqt1/assignment_loss‚2.046156463099689 2loss/FBt4/assignment_loss‚0.001198788150213659 0loss/SRqqt3/detection_loss‚2.910940627257029 1loss/SRqqt4/detection_loss‚1.0606831798032552 epoch‚1¶áŽƒžÈ‚(:“ ÌÜëÔÝÞÚÖ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.454' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.890800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vp®6É‚(:ö ÌÜëÔ…¸ØÛ$ memory_percent‚99.94800090789795& proc.memory.availableMB‚ 8.51953125 network.recv‚ 1.052051e+06% disk./.usageGB‚1179.0873718261719 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚62561.42529296875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 111.08984375 disk.sda2.out‚0( disk.nvme0c0n1.out‚62561.42529296875! disk.nvme0n1.in‚ 7729.87890625 disk.sda3.in‚0# proc.memory.rssMB‚ 3057.25390625 cpu‚25.321139813117327 network.sent‚ 4.670339e+06) disk./.usagePercent‚79.12954284345479 disk.sda.in‚0 disk.dm-1.out‚ 111.08984375# disk.nvme0c0n1.in‚ 7729.87890625) proc.memory.percent‚18.65999698638916 proc.cpu.threads‚17 disk.sda.out‚ 111.08984375 disk.dm-0.in‚ 7729.87890625 disk.sda2.in‚0 disk.drbd1260.out‚ 0.47265625& disk.nvme0n1.out‚62561.42529296875û´lžÊ‚(:“ ÛÜë԰ΖΠgpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.791000000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Ò!Ë‚(:ö ÛÜëÔ¾˜îÏ% disk./.usageGB‚1179.0873908996582 disk.sda2.out‚0 disk.drbd1260.out‚ 0.47265625# disk.nvme0c0n1.in‚ 7729.87890625! disk.nvme0n1.in‚ 7729.87890625 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.058826e+06 disk.dm-1.out‚ 111.49609375$ memory_percent‚99.94795322418213) proc.memory.percent‚18.65999698638916 proc.cpu.threads‚17) disk./.usagePercent‚79.12954412349247 disk.sda2.in‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚62744.35888671875& disk.nvme0n1.out‚62744.35888671875 disk.sda3.out‚ 111.49609375 disk.sda.out‚ 111.49609375# proc.memory.rssMB‚ 3057.25390625 network.sent‚ 4.779184e+06 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 7729.87890625# disk.dm-0.out‚62744.35888671875 disk.dm-1.in‚0& proc.memory.availableMB‚ 8.52734375 cpu‚25.320397765447865é OÑ“Ì‚(:ˆ êÜëÔ£ÿîË gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.851' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+d—ڀ͂(:õ êÜëÔʪŽÏ disk.drbd1260.in‚0( disk.nvme0c0n1.out‚63430.73876953125! disk.nvme0n1.in‚ 7729.87890625 disk.sda1.out‚0& proc.memory.availableMB‚ 8.52734375 disk.dm-0.in‚ 7729.87890625# disk.dm-0.out‚63430.73876953125 disk.dm-1.out‚ 112.01171875 disk.sda2.out‚0& disk.nvme0n1.out‚63430.73876953125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 network.sent‚ 4.786166e+06 disk.dm-1.in‚0$ memory_percent‚99.94795322418213# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 network.recv‚ 1.062395e+06 disk.drbd1260.out‚ 0.4765625# disk.nvme0c0n1.in‚ 7729.87890625 disk.sda3.out‚ 112.01171875 disk.sda.out‚ 112.01171875 cpu‚25.319563737976008 proc.cpu.threads‚17) disk./.usagePercent‚79.12954540353013% disk./.usageGB‚1179.0874099731445 disk.sda2.in‚0Ûþ§´“΂(:ˆ ùÜëÔÓÓöË gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÀfÏ‚(:ô ùÜëÔÎã¶Ð! disk.nvme0n1.in‚ 7894.69921875 disk.sda.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 4.789453e+06% disk./.usageGB‚1179.0873908996582 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 7894.69921875& disk.nvme0n1.out‚63752.29736328125 disk.sda3.out‚ 112.37890625 disk.sda1.out‚0 disk.dm-1.out‚ 112.37890625 disk.drbd1260.in‚0 disk.sda3.in‚0$ memory_percent‚99.94797706604004% proc.memory.availableMB‚ 8.5234375# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 cpu‚25.318705427233322 network.recv‚ 1.066228e+06 disk.sda2.out‚0 disk.drbd1260.out‚ 0.4765625 disk.sda.out‚ 112.37890625 disk.dm-0.in‚ 7894.69921875# disk.dm-0.out‚63752.29736328125 disk.dm-1.in‚0) disk./.usagePercent‚79.12954412349247( disk.nvme0c0n1.out‚63752.29736328125œ|‘î2Ð- _wandbruntime‚3294  _runtime‚3294Ø.¤ÄѾ _step‚183 % lr-AdamW/pg1‚0.001644102771486138 % lr-AdamW/pg2‚0.001644102771486138 trainer/global_step‚2299 $ _timestamp‚1.788538499471673e+09 _runtime‚3294.782556212âà©ÉÒà % lr-AdamW/pg1‚0.001644102771486138 % lr-AdamW/pg2‚0.001644102771486138 trainer/global_step‚2299 $ _timestamp‚1.788538499471673e+09 _runtime‚3294.782556212 _step‚183·è ¤2Ó- _wandbruntime‚3294  _runtime‚3294`gGò'ÔìO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2611442221568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187539907710461 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105165987 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193777633560576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2732829391360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42140397 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2600761016845 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2611442221568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42141020 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191166191339008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213817795948544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47353780 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47353782 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191166173336064 % _timestamp‚1.7885384994726317e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193776705587200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28358954 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2610532251136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57838549 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47353780 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193776705587200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47353782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184939146693616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191166191339008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2733053980160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211084966557184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0\Ûª÷'ÕñO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2611442221568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42141020 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191166191339008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213817795948544 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47353780 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47353782 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191166173336064 % _timestamp‚1.7885384994726317e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193776705587200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28358954 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2610532251136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57838549 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47353780 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193776705587200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47353782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184939146693616 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191166191339008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2733053980160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211084966557184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2611442221568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚187539907710461 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105165987 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193777633560576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588800 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2732829391360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42140397 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2600761016845 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144¸¤™Ãi2Ö- _wandbruntime‚3295  _runtime‚3295d–÷ð+×õM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193860374119936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57863406 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185018181327520 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57837038 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191247792421888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187620967592847 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105237348 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185018164288160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2601878290123 _runtime‚3295.796401392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42158233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213909451501056 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105210978 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2734235299328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186880 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191247810424832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191247810424832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105210978 trainer/global_step‚2299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57837038 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211175192261632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47373940 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2612563695104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2612563695104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57863406 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211175216201728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47373940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13787834 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚187620042578283 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 PëðOÐD>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788538500484518e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193859446133248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47373942 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2734019112448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47373942 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191247810424832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191247810424832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240126976 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105210978 trainer/global_step‚2299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57837038 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211175192261632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47373940 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2612563695104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2612563695104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57863406 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211175216201728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47373940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13787834 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚187620042578283 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788538500484518e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193859446133248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47373942 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2734019112448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47373942 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57863406 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185018181327520 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57837038 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191247792421888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187620967592847 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105237348 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185018164288160 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2601878290123 _runtime‚3295.796401392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42158233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213909451501056 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712¹A5NÒ2Ù- _wandbruntime‚3295  _runtime‚3295’fAWÚÑ /loss/FRt3/assignment_loss‚4.038347782912077 1loss/FBt1/assignment_loss‚0.35878826731555713 /loss/FRt2/detection_loss‚2.3971641468505065 _runtime‚3295.797874339 /loss/FRt2/assignment_loss‚3.356472631295522 1loss/SRqqt3/assignment_loss‚3.096699733734131 0loss/FBt3/assignment_loss‚0.3664148897963607 0loss/FBt4/assignment_loss‚1.3344233933836223 trainer/global_step‚2299 /loss/FBt2/detection_loss‚1.0063479576461218 epoch‚1 0loss/FRt1/assignment_loss‚3.8803605559544687 2loss/SRqqt4/assignment_loss‚1.3198282987321333 .loss/FRt4/detection_loss‚2.046934029533683 .loss/FBt3/detection_loss‚2.550786667399936 /loss/FRt4/assignment_loss‚2.789023441076279 1loss/SRqqt4/detection_loss‚0.9947675916097932 /loss/FBt1/detection_loss‚2.9714881953071144 /loss/FRt1/detection_loss‚2.6679945358863244 0loss/SRqqt3/detection_loss‚3.055462439060211 .loss/FBt4/detection_loss‚5.450174307823181 % _timestamp‚1.7885385004869308e+09 2loss/SRqqt1/assignment_loss‚1.8117194679137822 0loss/FBt2/assignment_loss‚0.2906236257737253 /loss/FRt3/detection_loss‚3.2432979764761747 0loss/SRqqt1/detection_loss‚2.116890792911117 1loss/SRqqt2/detection_loss‚2.8378303214907645 %loss/total_loss‚56.00465184094291 2loss/SRqqt2/assignment_loss‚2.0228107910603286 _step‚186èÂX\ÛÖ /loss/FRt2/assignment_loss‚3.356472631295522 1loss/SRqqt3/assignment_loss‚3.096699733734131 0loss/FBt3/assignment_loss‚0.3664148897963607 0loss/FBt4/assignment_loss‚1.3344233933836223 trainer/global_step‚2299 /loss/FBt2/detection_loss‚1.0063479576461218 epoch‚1 0loss/FRt1/assignment_loss‚3.8803605559544687 2loss/SRqqt4/assignment_loss‚1.3198282987321333 .loss/FRt4/detection_loss‚2.046934029533683 .loss/FBt3/detection_loss‚2.550786667399936 /loss/FRt4/assignment_loss‚2.789023441076279 1loss/SRqqt4/detection_loss‚0.9947675916097932 /loss/FBt1/detection_loss‚2.9714881953071144 /loss/FRt1/detection_loss‚2.6679945358863244 0loss/SRqqt3/detection_loss‚3.055462439060211 .loss/FBt4/detection_loss‚5.450174307823181 % _timestamp‚1.7885385004869308e+09 2loss/SRqqt1/assignment_loss‚1.8117194679137822 0loss/FBt2/assignment_loss‚0.2906236257737253 /loss/FRt3/detection_loss‚3.2432979764761747 0loss/SRqqt1/detection_loss‚2.116890792911117 1loss/SRqqt2/detection_loss‚2.8378303214907645 %loss/total_loss‚56.00465184094291 2loss/SRqqt2/assignment_loss‚2.0228107910603286 _step‚186 /loss/FRt3/assignment_loss‚4.038347782912077 1loss/FBt1/assignment_loss‚0.35878826731555713 /loss/FRt2/detection_loss‚2.3971641468505065 _runtime‚3295.797874339ºô<ù“Ü‚(:ˆ ˆÝëÔÚÉüÍ gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.602' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¢ ÊÝ‚(:ô ˆÝëÔÕ×àÐ disk.sda.out‚ 112.87890625% disk./.usageGB‚1179.0873336791992! disk.nvme0n1.in‚ 7894.69921875 disk.sda3.in‚0 disk.sda3.out‚ 112.87890625 disk.dm-0.in‚ 7894.69921875 disk.drbd1260.out‚ 0.48046875 cpu‚25.317868905410617 network.sent‚ 4.795456e+06# disk.nvme0c0n1.in‚ 7894.69921875 disk.sda1.in‚0# disk.dm-0.out‚63901.05029296875 disk.dm-1.out‚ 112.87890625 disk.sda2.out‚0 disk.drbd1260.in‚0$ memory_percent‚99.94821548461914) disk./.usagePercent‚79.12954028337946( disk.nvme0c0n1.out‚63901.05029296875& disk.nvme0n1.out‚63901.05029296875 disk.sda.in‚0 disk.dm-1.in‚0$ proc.memory.availableMB‚8.484375# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 network.recv‚ 1.070646e+06 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17Š‡c“Þ‚(:ˆ —ÝëÔÕÖ®Ê gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l\€¤zß‚(:ï —ÝëÔ¸¨¬Ì# disk.dm-0.out‚64174.59716796875$ memory_percent‚99.94831085205078 network.sent‚ 4.90431e+06) disk./.usagePercent‚79.12954412349247 disk.sda2.in‚0 disk.drbd1260.out‚ 0.48046875 disk.sda3.out‚ 113.7890625 disk.sda.in‚0# proc.memory.availableMB‚8.46875# proc.memory.rssMB‚ 3057.25390625! disk.nvme0n1.in‚ 7894.69921875% disk./.usageGB‚1179.0873908996582 disk.dm-1.out‚ 113.7890625# disk.nvme0c0n1.in‚ 7894.69921875 disk.sda1.out‚0 cpu‚25.316966446642176 proc.cpu.threads‚17& disk.nvme0n1.out‚64174.59716796875 disk.sda3.in‚0 disk.sda.out‚ 113.7890625 disk.sda1.in‚0) proc.memory.percent‚18.65999698638916 network.recv‚ 1.077162e+06 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚64174.59716796875 disk.dm-0.in‚ 7894.69921875³¾=sà‚(:è ¦ÝëÔãˬÏ% disk.nvme0c0n1.out‚64506.05078125) disk./.usagePercent‚79.12797044517949 disk.dm-1.out‚ 114.12109375 disk.sda2.in‚0 disk.drbd1260.out‚0.484375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.94983673095703% disk./.usageGB‚1179.0639419555664 disk.sda2.out‚0 cpu‚25.316104639216622 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 7894.69921875 network.recv‚ 1.080637e+06 disk.dm-0.out‚64506.05078125 disk.dm-1.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 7894.69921875# disk.nvme0n1.out‚64506.05078125 disk.sda3.in‚0 network.sent‚ 4.909388e+06 disk.dm-0.in‚ 7894.69921875 disk.sda3.out‚ 114.12109375 disk.sda.in‚0 disk.sda.out‚ 114.12109375# proc.memory.availableMB‚8.21875# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.6599969863891669VÙ“á‚(:ˆ ¦ÝëÔþúÑÎ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.081' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç?yâ‚(:’ µÝëÔØçÏÌ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.65' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.330000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“4 ómã‚(:â µÝëÔ´¬Ï disk.drbd1260.in‚0 disk.nvme0c0n1.in‚ 8045.03125% disk.nvme0c0n1.out‚65019.93359375 disk.nvme0n1.in‚ 8045.03125# disk.nvme0n1.out‚65019.93359375% disk./.usageGB‚1179.0873908996582 disk.sda1.out‚0 disk.dm-0.in‚ 8045.03125 disk.dm-0.out‚65019.93359375 disk.sda3.in‚0& proc.memory.availableMB‚ 8.46484375 cpu‚25.315443847808893 network.recv‚ 1.084366e+06 disk.sda.out‚ 114.50390625 disk.sda2.in‚0 disk.drbd1260.out‚0.484375 disk.sda3.out‚ 114.50390625# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 network.sent‚ 4.914485e+06) disk./.usagePercent‚79.12954412349247 disk.dm-1.in‚0 disk.sda2.out‚0$ memory_percent‚99.94833469390869 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 114.50390625CóGe2ä- _wandbruntime‚3356  _runtime‚3356ݯƒÆåÀ _runtime‚3356.137825931 _step‚187 & lr-AdamW/pg1‚0.0016402535529048193 & lr-AdamW/pg2‚0.0016402535529048193 trainer/global_step‚2349 $ _timestamp‚1.788538560826803e+09jÇYÿËæÅ _runtime‚3356.137825931 _step‚187 & lr-AdamW/pg1‚0.0016402535529048193 & lr-AdamW/pg2‚0.0016402535529048193 trainer/global_step‚2349 $ _timestamp‚1.788538560826803e+09»7„Þ/2ç- _wandbruntime‚3356  _runtime‚3356*Ìò'èìO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59083451 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48361780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14072341 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195247127627264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215596275719680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2657546246949 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28962206 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2791708597248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2666619676160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191547589707125 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59057109 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚195247145630208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59083451 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188890043460176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2656638271953 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43035182 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 trainer/global_step‚2349 % _timestamp‚1.7885385608282073e+09 _step‚188 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28962204 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14072976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2666619676160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48361782 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218387960376832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197913747303424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215596251779584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191546664692769 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195247127627264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2667529646592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2791935283200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107418889 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107445233 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197913747303424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48361782 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48361780 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43034545 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188890026420816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚3356.139390258 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2667529646592 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615ŒFØ°éñO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107445233 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197913747303424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48361782 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48361780 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43034545 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188890026420816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _runtime‚3356.139390258 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2667529646592 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59083451 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48361780 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14072341 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195247127627264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215596275719680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2657546246949 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907974996 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28962206 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2791708597248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2666619676160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191547589707125 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59057109 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚195247145630208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59083451 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226685952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188890043460176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2656638271953 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43035182 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 trainer/global_step‚2349 % _timestamp‚1.7885385608282073e+09 _step‚188 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28962204 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14072976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2666619676160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48361782 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218387960376832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197913747303424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215596251779584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191546664692769 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195247127627264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2667529646592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2791935283200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107418889¼e{òC2ê- _wandbruntime‚3357  _runtime‚3357.ÿ{¬û&ëõM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107490250 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107463880 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59108308 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195328764716032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218479603789824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188969044015360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14078811 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2793102365696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48381940 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107463880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59081940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28974269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28974271 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107490250 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2667741136384 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚189 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197997415836160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197996487849472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195328746713088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43052468 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191627724575155 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218479353177088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195328746713088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197997415836160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚215686501424128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2792875693056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2668651120128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 $ _timestamp‚1.788538562156138e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215686477484032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2657755545231 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188969061054720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48381942 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43053082 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2668651120128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191626799560591 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48381942 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚189 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197997415836160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197996487849472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195328746713088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43052468 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191627724575155 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218479353177088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195328746713088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197997415836160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚215686501424128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2792875693056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2668651120128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 $ _timestamp‚1.788538562156138e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215686477484032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2657755545231 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188969061054720 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48381942 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43053082 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2668651120128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191626799560591 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48381942 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107490250 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107463880 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59108308 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195328764716032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218479603789824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188969044015360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909983744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14078811 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2793102365696 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48381940 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107463880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59081940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28974269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728385 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144½žº˜Y2í- _wandbruntime‚3357  _runtime‚3357¤KqoRîÌ epoch‚1 /loss/FRt4/detection_loss‚1.8937389186670308 0loss/SRqqt4/detection_loss‚1.177134117581577 .loss/FBt1/detection_loss‚2.263644651130394 0loss/FBt4/assignment_loss‚1.2937456307311852 /loss/FRt2/assignment_loss‚2.906316365256454 /loss/FRt3/assignment_loss‚4.304700578962054 /loss/FBt2/detection_loss‚0.9816425266637377 _step‚190 1loss/FBt2/assignment_loss‚0.25580625555338204 0loss/FBt3/assignment_loss‚0.5864836607126336 1loss/SRqqt1/detection_loss‚1.8893483579158783 /loss/FRt4/assignment_loss‚2.433586257790762 2loss/SRqqt1/assignment_loss‚2.1732404686155773 %loss/total_loss‚55.37155564905027 _runtime‚3357.471199174 .loss/FRt1/assignment_loss‚4.62360401691929 .loss/FRt2/detection_loss‚2.083776051122131 1loss/SRqqt3/assignment_loss‚2.982238457753108 /loss/FRt3/detection_loss‚2.9017814431871685 trainer/global_step‚2349 % _timestamp‚1.7885385621602864e+09 .loss/FRt1/detection_loss‚3.011986765169328 .loss/FBt4/detection_loss‚4.755571564038594 1loss/SRqqt2/assignment_loss‚1.995458549509446 2loss/SRqqt4/assignment_loss‚1.3081726739183068 0loss/FBt1/assignment_loss‚0.5485616514730888 1loss/SRqqt2/detection_loss‚2.4448034477730594 0loss/SRqqt3/detection_loss‚4.247831514248481 .loss/FBt3/detection_loss‚2.308381724357605Õ '×ïÑ .loss/FRt1/detection_loss‚3.011986765169328 .loss/FBt4/detection_loss‚4.755571564038594 1loss/SRqqt2/assignment_loss‚1.995458549509446 2loss/SRqqt4/assignment_loss‚1.3081726739183068 0loss/FBt1/assignment_loss‚0.5485616514730888 1loss/SRqqt2/detection_loss‚2.4448034477730594 0loss/SRqqt3/detection_loss‚4.247831514248481 .loss/FBt3/detection_loss‚2.308381724357605 epoch‚1 /loss/FRt4/detection_loss‚1.8937389186670308 0loss/SRqqt4/detection_loss‚1.177134117581577 .loss/FBt1/detection_loss‚2.263644651130394 0loss/FBt4/assignment_loss‚1.2937456307311852 /loss/FRt2/assignment_loss‚2.906316365256454 /loss/FRt3/assignment_loss‚4.304700578962054 /loss/FBt2/detection_loss‚0.9816425266637377 _step‚190 1loss/FBt2/assignment_loss‚0.25580625555338204 0loss/FBt3/assignment_loss‚0.5864836607126336 1loss/SRqqt1/detection_loss‚1.8893483579158783 /loss/FRt4/assignment_loss‚2.433586257790762 2loss/SRqqt1/assignment_loss‚2.1732404686155773 %loss/total_loss‚55.37155564905027 _runtime‚3357.471199174 .loss/FRt1/assignment_loss‚4.62360401691929 .loss/FRt2/detection_loss‚2.083776051122131 1loss/SRqqt3/assignment_loss‚2.9822384577a÷B €53108 /loss/FRt3/detection_loss‚2.9017814431871685 trainer/global_step‚2349 % _timestamp‚1.7885385621602864e+09¾½ –’ð‚(:‡ ÄÝëÔÚ ®Ï gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|=¾;mñ‚(:â ÄÝëÔ—€Ó( disk.nvme0c0n1.out‚65082.79345703125 disk.sda.out‚ 114.90625 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚65082.79345703125# memory_percent‚99.9488115310669# proc.memory.rssMB‚ 3057.25390625 network.recv‚ 1.089014e+06 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 8045.03125 disk.dm-1.in‚0 disk.dm-1.out‚ 114.90625& proc.memory.availableMB‚ 8.38671875 cpu‚25.31466016071287 network.sent‚ 4.920374e+06% disk./.usageGB‚1179.0873336791992 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.48828125 disk.nvme0n1.in‚ 8045.03125 disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚79.12954028337946 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 8045.03125& disk.nvme0n1.out‚65082.79345703125 disk.sda3.out‚ 114.90625) proc.memory.percent‚18.65999698638916vFu8“ò‚(:ˆ ÓÝëÔ•¹ØË gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MsÒ(uó‚(:ê ÓÝëÔõÈ›Ï) disk./.usagePercent‚79.12954412349247 disk.drbd1260.out‚ 0.48828125& disk.nvme0n1.out‚65672.97705078125 disk.sda3.out‚ 115.5390625$ memory_percent‚99.94871616363525# proc.memory.rssMB‚ 3057.25390625) proc.memory.percent‚18.65999698638916 cpu‚25.313923231829225 network.sent‚ 5.029512e+06% disk./.usageGB‚1179.0873908996582 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 8.40234375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 115.5390625 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚65672.97705078125 disk.nvme0n1.in‚ 8045.03125 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.095672e+06 disk.nvme0c0n1.in‚ 8045.03125 disk.dm-0.in‚ 8045.03125# disk.dm-0.out‚65672.97705078125 disk.dm-1.in‚0 disk.dm-1.out‚ 115.5390625¯Ur™“ô‚(:ˆ âÝëÔ’øÑÌ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬·;(sõ‚(:è âÝëÔò´îÒ disk.sda.in‚0 cpu‚25.313152940799302 network.sent‚ 5.034357e+06 disk.sda2.out‚0' disk.nvme0c0n1.out‚65764.0244140625 disk.sda1.in‚0 disk.dm-0.in‚ 8045.03125 disk.dm-1.out‚ 116.33203125$ memory_percent‚99.94871616363525 network.recv‚ 1.09863e+06 disk.nvme0c0n1.in‚ 8045.03125% disk.nvme0n1.out‚65764.0244140625 disk.sda.out‚ 116.33203125" disk.dm-0.out‚65764.0244140625) proc.memory.percent‚18.65999698638916) disk./.usagePercent‚79.12954565953767% disk./.usageGB‚1179.0874137878418 disk.drbd1260.out‚ 0.4921875 disk.sda1.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 8.40234375# proc.memory.rssMB‚ 3057.25390625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 8045.03125 disk.sda3.in‚0 disk.sda3.out‚ 116.33203125˜©À‘ö‚(:† ñÝëÔ‡”Ê gpu.0.gpu‚34.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚118.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û+å»}÷‚(:ò ñÝëÔŽöÒÍ disk.sda2.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚66119.52392578125 disk.sda3.out‚ 116.66015625$ memory_percent‚99.94878768920898 cpu‚25.31230592594639% disk./.usageGB‚1179.0873947143555 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 8052.73046875( disk.nvme0c0n1.out‚66119.52392578125! disk.nvme0n1.in‚ 8052.73046875 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 116.66015625 network.sent‚ 5.038746e+06 network.recv‚ 1.101939e+06) disk./.usagePercent‚79.12954437949999 disk.dm-0.in‚ 8052.73046875 disk.sda1.in‚0 disk.sda1.out‚0$ proc.memory.availableMB‚8.390625) proc.memory.percent‚18.65999698638916# disk.dm-0.out‚66119.52392578125 disk.dm-1.out‚ 116.66015625 disk.sda2.out‚0 disk.drbd1260.out‚ 0.4921875# proc.memory.rssMB‚ 3057.25390625 proc.cpu.threads‚17µ{¯ä2ø- _wandbruntime‚3418  _runtime‚3418<þŒ5ÇùÁ _step‚191 & lr-AdamW/pg1‚0.0016362381350172562 & lr-AdamW/pg2‚0.0016362381350172562 trainer/global_step‚2399 % _timestamp‚1.7885386236044548e+09 _runtime‚3418.915574713Sà™eÌúÆ % _timestamp‚1.7885386236044548e+09 _runtime‚3418.915574713 _step‚191 & lr-AdamW/pg1‚0.0016362381350172562 & lr-AdamW/pg2‚0.0016362381350172562 trainer/global_step‚2399¿Á 6®2û- _wandbruntime‚3418  _runtime‚3418äL&öò'üìO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49369782 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2850778056704 % _timestamp‚1.7885386236058629e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2723617071616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192840923187376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109698135 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195553421675077 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220107560942080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43930538 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60328353 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚202050789019648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218297344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49369780 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60302011 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2722707101184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14365082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14364465 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43929919 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199328099921408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2723617071616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60328353 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2722707101184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49369780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202050789019648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199328081918464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _runtime‚3418.917048193 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195554346689433 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192840906148016 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109671791 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222958338998784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2850559759360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2713423502057 trainer/global_step‚2399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222958096761344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199328099921408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 141357373441ÑÞ’÷'ýñO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60302011 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29565454 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60302011 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2722707101184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14365082 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14364465 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43929919 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014356 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199328099921408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2723617071616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728385 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565456 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60328353 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 329146880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2722707101184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49369780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202050789019648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199328081918464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _runtime‚3418.917048193 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195554346689433 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192840906148016 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _step‚192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109671791 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222958338998784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2850559759360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2713423502057 trainer/global_step‚2399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚222958096761344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199328099921408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49369782 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2850778056704 % _timestamp‚1.7885386236058629e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2723617071616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192840923187376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109698135 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416513 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚195553421675077 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220107560942080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43930538 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60328353 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚202050789019648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218297344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49369780 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202134457552384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14370323 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220197762706432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2714540775543 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199409701004288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60326842 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60353210 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43948486 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202134457552384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14370965 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2723828561408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49389940 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43947842 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223049483466752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2724738545152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49389942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109716782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195633556542899 # _timestamp‚1.78853862480252e+09 P65696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60326842 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909983744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49389942 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109743152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚193 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927986688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109743152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254807040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 _step‚193 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975204 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202134457552384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14370323 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220197762706432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2714540775543 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199409701004288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 329146880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014564 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60326842 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927986688 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60353210 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43948486 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202134457552384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14370965 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2723828561408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49389940 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43947842 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223049483466752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2724738545152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49389942 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416513 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109716782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195633556542899 # _timestamp‚1.78853862480252e+09 PDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚2417 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚50511144 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚206306591706624 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2768944711168 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038208 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30263227 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚199571729122163 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20709 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2768261672960 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111743044 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚224578935522304 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 trainer/global_step‚2417 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚50511144 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767168 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚206306591706624 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2768944711168 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038208 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30263227 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚199571729122163 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20709 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2768261672960 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13205766144ÃÏ&*.2“- _wandbruntime‚3509  _runtime‚3509@oìc)”ÝR ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 196539904 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚224635227016192 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2759080695319 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737344 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2768751523840 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2897634024960 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701041152 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106800 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚203591907902464 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚50526622 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2768751523840 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚45017301 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚227533033640960 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2758399627879 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728385 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚203591889899520 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885387137431555e+09 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚14744291 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2897806624768 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚196863545703552 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683038208 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚30272791 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206361342464512 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111767334 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111746623 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚203591889899520 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚61220003 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30272789 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚196863562742912 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚3509.054773205 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚61240712 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067440 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚61220003 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚14744510 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227532837101056 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2769434562048 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20709 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32004 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2769434562048 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚199621945331431 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 _step‚196 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767168 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111767334 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚50526620 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚61240712 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32247 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 172599808 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737344 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701041152 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚224635203076096 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚199622643438231 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701041152 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚224635203076096 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚199622643438231 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 196539904 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚224635227016192 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2759080695319 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737344 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2768751523840 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2897634024960 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚615 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701041152 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106800 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚203591907902464 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚50526622 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2768751523840 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚45017301 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚227533033640960 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2758399627879 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1187728385 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14401142784 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚203591889899520 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885387137431555e+09 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚14744291 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2897806624768 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚196863545703552 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 683038208 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚30272791 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206361342464512 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 111767334 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111746623 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚2035918898995dº/dõ20 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚61220003 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30272789 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚196863562742912 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚3509.054773205 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚219 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚61240712 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067440 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚61220003 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚14744510 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227532837101056 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2769434562048 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14401142784 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1195376640ÄýÙ:“–‚(:ˆ ÚÞëÔÕÁÏ gpu.0.gpu‚47.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ4|—‚(:ñ ÚÞëÔ„ÛÓÓ disk.dm-1.in‚0' disk.nvme0c0n1.out‚68362.5498046875% proc.memory.availableMB‚ 6.9140625 network.recv‚ 1.133954e+06) disk./.usagePercent‚79.08431783249358 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 120.15234375 disk.sda2.in‚0 disk.drbd1260.out‚ 0.5078125# disk.nvme0c0n1.in‚ 8308.20703125 network.sent‚ 5.261513e+06% disk.nvme0n1.out‚68362.5498046875 disk.sda1.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.95779991149902# proc.memory.rssMB‚ 3065.31640625! disk.nvme0n1.in‚ 8308.20703125 disk.sda3.out‚ 120.15234375" disk.dm-0.out‚68362.5498046875 disk.sda2.out‚0* proc.memory.percent‚18.709206581115723 cpu‚25.403468578023404 proc.cpu.threads‚17$ disk./.usageGB‚1178.413486480713 disk.sda.out‚ 120.15234375 disk.sda1.out‚0 disk.dm-0.in‚ 8308.20703125 ¹»’˜‚(:‡ éÞëÔÝÊË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DâÆ>™‚(:ö éÞëÔ¦÷öË disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 120.41015625) disk./.usagePercent‚79.07708101151567# disk.nvme0c0n1.in‚ 8308.20703125! disk.nvme0n1.in‚ 8308.20703125 disk.sda1.in‚0# disk.dm-0.out‚68647.19580078125& proc.memory.availableMB‚ 7.01171875* proc.memory.percent‚18.709206581115723% disk./.usageGB‚1178.3056526184082 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚68647.19580078125 disk.dm-0.in‚ 8308.20703125 disk.dm-1.in‚0$ memory_percent‚99.95720386505127# proc.memory.rssMB‚ 3065.31640625 network.sent‚ 5.369201e+06 network.recv‚ 1.139585e+06( disk.nvme0c0n1.out‚68647.19580078125 disk.sda3.out‚ 120.41015625 disk.sda.out‚ 120.41015625 disk.sda1.out‚0 cpu‚25.425885589692466 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1260.out‚ 0.5078125jG:2š- _wandbruntime‚3526  _runtime‚3526–ˆç†n›–§ 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9573997855186462 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _step‚197 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.21720032393932343 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5239822864532471 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.308908075094223 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9140212535858154 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2441035658121109 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2843005955219269 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19252872467041016 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.309266597032547 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 $ _timestamp‚1.788538731122245e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13508224487304688 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3861808776855469 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8533333539962769 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.28922927379608154 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966834783554077 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.29610222578048706 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.17801451683044434 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19753116369247437 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03697215020656586 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 jet/accuracy_4_of_5‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2323126643896103 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9699811935424805 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_2_of_5‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10132373869419098 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0574130117893219 *jet/accuracy_1_of_3‚0.7759923338890076 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6255968809127808 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005794910714030266 jet/accuracy_1_of_7‚1 +"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.25 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.25905898213386536 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19570490717887878 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9780400991439819 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9735848903656006 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018669690936803818 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.19230669736862183 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2938392162322998 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 +jet/accuracy_3_of_6‚0.11538461595773697 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30937495827674866 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32754406332969666 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0527433343231678 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2906533181667328 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.16002723574638367 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_3_of_5‚0.0396377369761467 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8928571343421936 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027508962899446487 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.57916659116745 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.3264337182044983 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07145832479000092 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.31624990701675415 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3802623450756073 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3274804353713989 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.36034873127937317 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12630651891231537 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3470824360847473 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19342787563800812 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04382184147834778 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.003843106562271714 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9645880460739136 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4186182916164398 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5726075172424316 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1880694329738617 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.20594541728496552 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18773536384105682 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9957606792449951 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45253822207450867 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.29643240571022034 8"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.226666659116745 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32232144474983215 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.008824244141578674 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_4_of_4‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5001479387283325 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9679487943649292 jet/accuracy_7_of_7‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8090403079986572 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28682297468185425 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20503513514995575 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9937856793403625 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0011337868636474013 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3579951822757721 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2291666716337204 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2710728049278259 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.048246245831251144 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 _runtime‚3526.435910685 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6247876882553101 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9725618362426758 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9731996059417725 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9848381280899048 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 jet/accuracy_2_of_7‚0 epoch‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2154398262500763 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.935119092464447 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3201501667499542 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.06774770468473434 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3712824583053589 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.26452505588531494 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 particle/accuracy_1_of_3‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.25 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9838138222694397 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9886904358863831 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 jet/accuracy_6_of_7‚0 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.98922199010849 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_1_of_5‚0.8790450692176819 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3340569734573364 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5578404664993286 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9892205595970154 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.28615203499794006 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9344724416732788 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.19729730486869812 particle/accuracy_3_of_3‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5174466371536255 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011895342729985714 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9612069725990295 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9666706919670105 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9079861044883728 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3749999701976776 ,jet/accuracy_3_of_3‚0.015867188572883606 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9784724116325378 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9340917468070984 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2266598641872406 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_7‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3024161756038666 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.27903708815574646 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2209305465221405 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.26016098260879517 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2824074327945709 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.54365074634552 particle/accuracy_1_of_5‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9776278734207153 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_4‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2776259779930115 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.336456298828125 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.958509087562561 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.25823044776916504 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4385964870452881 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17714887857437134 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2741343080997467 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004232804290950298 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9340739250183105 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3394771218299866 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.25034528970718384 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11273499578237534 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.28197672963142395 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002521633170545101 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.29385966062545776 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23607780039310455 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.965330958366394 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 (particle/accuracy‚0.8153107762336731 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.28152742981910706 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5151716470718384 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9030734300613403 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22830435633659363 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3198555111885071 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8874268531799316 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 *loss/val_total_loss‚55.024328550925624 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3024161756038666 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.014192388392984867 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9866322875022888 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07617095857858658 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20293211936950684 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2669753134250641 particle/accuracy_5_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966834783554077 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9539265036582947 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5500832200050354 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3141072690486908 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6556950211524963 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2697843909263611 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5459961891174316 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.18132714927196503 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18614013493061066 jet/accuracy_5_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3095678687095642 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3888888955116272 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2275121808052063 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5127118229866028 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.38353943824768066 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.203603595495224 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 particle/accuracy_2_of_6‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2231515794992447 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"PuritnëDý%y/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21597376465797424 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2520114779472351 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20590150356292725 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22968854010105133 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9841346144676208 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2363496571779251 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.26851025223731995 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.004282019566744566 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23248741030693054 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.181547611951828 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 particle/accuracy_6_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.553205132484436 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3630562424659729 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.288507342338562 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05564069375395775 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2083333283662796 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9521570205688477 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4435344636440277 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 6 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.968992292881012 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2372286170721054 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9557945728302002 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3660503625869751 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.28240740299224854 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11300324648618698 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21391792595386505 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9804903864860535 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12013723701238632 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.8235475420951843 particle/accuracy_2_of_3‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.18126364052295685 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5284722447395325 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3263888657093048 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12304434180259705 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0557556115090847 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.22921450436115265 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2651515305042267 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9222818613052368 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4166666567325592 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9802631735801697 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2098734825849533 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3518519103527069 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.33201947808265686 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5389183759689331 jet/accuracy_5_of_5‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 jet/accuracy_5_of_6‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5131925344467163 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18812642991542816 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22060465812683105 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2633090019226074 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9664269089698792 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011616304516792297 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18518517911434174 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3783000111579895 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25295212864875793 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0019930850248783827 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3815728425979614 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.0962335467338562 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.010362065397202969 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9251735806465149 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.34218630194664 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9633982181549072 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9690955877304077 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12013723701238632 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23814645409584045 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.18501685559749603 particle/accuracy_2_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2169191837310791 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0595238097012043 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18598200380802155 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4861394166946411 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13017703592777252 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 trainer/global_step‚2417 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18273666501045227 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006581461522728205 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.27105262875556946 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1732289344072342 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17327789962291718 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.30878299474716187 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0ºnýÛZœ›§ 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23607780039310455 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.965330958366394 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 (particle/accuracy‚0.8153107762336731 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.28152742981910706 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5151716470718384 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9030734300613403 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22830435633659363 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_4‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3198555111885071 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8874268531799316 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 *loss/val_total_loss‚55.024328550925624 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3024161756038666 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.014192388392984867 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9866322875022888 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07617095857858658 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20293211936950684 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2669753134250641 particle/accuracy_5_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966834783554077 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9539265036582947 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5500832200050354 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3141072690486908 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.6556950211524963 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2697843909263611 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5459961891174316 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.18132714927196503 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18614013493061066 jet/accuracy_5_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3095678687095642 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3888888955116272 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2275121808052063 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5127118229866028 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.38353943824768066 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.203603595495224 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 particle/accuracy_2_of_6‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2231515794992447 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21597376465797424 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2520114779472351 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20590150356292725 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22968854010105133 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9841346144676208 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2363496571779251 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.26851025223731995 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.004282019566744566 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.23248741030693054 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.181547611951828 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 particle/accuracy_6_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.553205132484436 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3630562424659729 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.288507342338562 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05564069375395775 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2083333283662796 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9521570205688477 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4435344636440277 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 6 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.968992292881012 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2372286170721054 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9557945728302002 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3660503625869751 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.28240740299224854 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11300324648618698 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.21391792595386505 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9804903864860535 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12013723701238632 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_4‚0.8235475420951843 particle/accuracy_2_of_3‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.18126364052295685 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5284722447395325 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3263888657093048 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12304434180259705 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0557556115090847 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.22921450436115265 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2651515305042267 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9222818613052368 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4166666567325592 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9802631735801697 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8333333134651184 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.2098734825849533 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3518519103527069 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.33201947808265686 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5389183759689331 jet/accuracy_5_of_5‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14444445073604584 jet/accuracy_5_of_6‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5131925344467163 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18812642991542816 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22060465812683105 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2633090019226074 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9664269089698792 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011616304516792297 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18518517911434174 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3783000111579895 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.25295212864875793 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0019930850248783827 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3815728425979614 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.0962335467338562 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.010362065397202969 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9251735806465149 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.34218630194664 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9633982181549072 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9690955877304077 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12013723701238632 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.23814645409584045 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.18501685559749603 particle/accuracy_2_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2169191837310791 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0595238097012043 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18598200380802155 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4861394166946411 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13017703592777252 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 trainer/global_step‚2417 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18273666501045227 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.006581461522728205 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.27105262875556946 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1732289344072342 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17327789962291718 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.30878299474716187 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9573997855186462 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _step‚197 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.21720032393932343 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5239822864532471 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.308908075094223 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9140212535858154 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2441035658121109 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2843005955219269 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.19252872467041016 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.309266597032547 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 $ _timestamp‚1.788538731122245e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13508224487304688 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3861808776855469 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8533333539962769 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.28922927379608154 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966834783554077 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.29610222578048706 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.17801451683044434 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19753116369247437 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03697215020656586 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 jet/accuracy_4_of_5‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2323126643896103 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9699811935424805 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 particle/accuracy_2_of_5‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10132373869419098 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0574130117893219 *jet/accuracy_1_of_3‚0.7759923338890076 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6255968809127808 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005794910714030266 jet/accuracy_1_of_7‚1 +"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.25 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.25905898213386536 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19570490717887878 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9780400991439819 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9735848903656006 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018669690936803818 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.19230669736862183 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2938392162322998 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 +jet/accuracy_3_of_6‚0.11538461595773697 particle/accuracy_4_of_7‚1 particle/accuracy_5_of_7‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30937495827674866 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.32754406332969666 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0527433343231678 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2906533181667328 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6742424368858337 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.16002723574638367 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 *jet/accuracy_3_of_5‚0.0396377369761467 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.8928571343421936 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027508962899446487 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.57916659116745 ,&Purit-9ÈßÇ8y/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.3264337182044983 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07145832479000092 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.31624990701675415 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3802623450756073 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3274804353713989 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.36034873127937317 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12630651891231537 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3470824360847473 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19342787563800812 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04382184147834778 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.003843106562271714 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9645880460739136 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4186182916164398 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5726075172424316 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1880694329738617 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.20594541728496552 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18773536384105682 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9957606792449951 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45253822207450867 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.29643240571022034 8"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.226666659116745 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32232144474983215 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.008824244141578674 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_4_of_4‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5001479387283325 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9679487943649292 jet/accuracy_7_of_7‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8090403079986572 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.28682297468185425 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20503513514995575 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9937856793403625 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0011337868636474013 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3579951822757721 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2291666716337204 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2710728049278259 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.048246245831251144 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 _runtime‚3526.435910685 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6247876882553101 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9725618362426758 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9731996059417725 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9848381280899048 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 jet/accuracy_2_of_7‚0 epoch‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2154398262500763 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.935119092464447 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3201501667499542 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.06774770468473434 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3712824583053589 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.26452505588531494 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 particle/accuracy_1_of_3‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.25 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9838138222694397 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9886904358863831 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 jet/accuracy_6_of_7‚0 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.98922199010849 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_1_of_5‚0.8790450692176819 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3340569734573364 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5578404664993286 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9892205595970154 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.28615203499794006 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9344724416732788 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.19729730486869812 particle/accuracy_3_of_3‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5174466371536255 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011895342729985714 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9612069725990295 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9666706919670105 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9079861044883728 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3749999701976776 ,jet/accuracy_3_of_3‚0.015867188572883606 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9784724116325378 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9340917468070984 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2266598641872406 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_7‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3024161756038666 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.27903708815574646 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2209305465221405 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.26016098260879517 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2824074327945709 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.54365074634552 particle/accuracy_1_of_5‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9776278734207153 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_4‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.06382978707551956 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.2776259779930115 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.336456298828125 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.958509087562561 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.25823044776916504 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4385964870452881 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17714887857437134 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2741343080997467 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.004232804290950298 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9340739250183105 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3394771218299866 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.25034528970718384 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.11273499578237534 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.28197672963142395 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002521633170545101 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.29385966062545776 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1ňVV“‚(:ˆ øÞëÔ‹ó¼Ë gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=œïxž‚(:í øÞëÔÑ­…Î proc.cpu.threads‚17# disk.dm-0.out‚70199.56396484375 disk.dm-1.out‚ 120.83203125 disk.drbd1260.out‚ 0.51171875 disk.nvme0n1.in‚ 8470.4609375& disk.nvme0n1.out‚70199.56396484375 disk.sda3.in‚0 disk.sda3.out‚ 120.83203125 disk.sda.in‚0% disk./.usageGB‚1178.9465942382812 disk.sda2.in‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚70199.56396484375$ memory_percent‚91.94450378417969% proc.memory.availableMB‚ 1319.8125 proc.memory.rssMB‚ 3073.53125 cpu‚25.42463154475106 network.sent‚ 5.37495e+06) disk./.usagePercent‚79.12009514141118 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 8470.4609375 disk.sda1.out‚0 network.recv‚ 1.143684e+06 disk.dm-0.in‚ 8470.4609375 disk.dm-1.in‚0 disk.sda.out‚ 120.83203125 disk.sda1.in‚0) proc.memory.percent‚18.75934600830078ŽE²î“Ÿ‚(:ˆ ‡ßëÔ¯äÉË gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼s|u} ‚(:ò ‡ßëÔëèõÏ disk.sda1.in‚0 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 network.recv‚ 1.157755e+06( disk./.usagePercent‚79.1200989815242 disk.dm-0.in‚ 8470.4609375# disk.dm-0.out‚70416.19287109375 disk.dm-1.in‚0 disk.dm-1.out‚ 121.49609375 disk.sda2.out‚0) proc.memory.availableMB‚ 1318.99609375 network.sent‚ 5.507456e+06% disk./.usageGB‚1178.9466514587402 disk.nvme0n1.in‚ 8470.4609375 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚91.94948673248291 cpu‚25.423252507302283 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.51171875" disk.nvme0c0n1.in‚ 8470.4609375( disk.nvme0c0n1.out‚70416.19287109375 disk.drbd1260.in‚0 disk.sda3.out‚ 121.49609375 disk.sda.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚70416.19287109375 disk.sda.out‚ 121.49609375“¢À“¡‚(:ˆ –ßëÔݹÈË gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚141.814' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3628 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d:³-{¢‚(:ð –ßëÔˆþóÍ disk.dm-1.out‚ 122.08203125" disk.nvme0c0n1.in‚ 8470.4609375) proc.memory.percent‚18.75934600830078 cpu‚25.422011029307793 proc.cpu.threads‚17 network.sent‚ 5.512527e+06 disk.dm-1.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 3073.53125 network.recv‚ 1.161105e+06 disk.sda.out‚ 122.08203125 disk.sda2.in‚0 disk.drbd1260.out‚0.515625 disk.nvme0n1.in‚ 8470.4609375& disk.nvme0n1.out‚70699.69677734375 disk.sda3.out‚ 122.08203125$ memory_percent‚91.94655418395996) disk./.usagePercent‚79.12010026156187 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚70699.69677734375 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚70699.69677734375 disk.sda3.in‚0( proc.memory.availableMB‚ 1319.4765625% disk./.usageGB‚1178.9466705322266 disk.sda.in‚0 disk.dm-0.in‚ 8470.4609375Æ]û2£- _wandbruntime‚3570  _runtime‚3570¢¢(0ǤÁ _runtime‚3570.731642262 _step‚198 & lr-AdamW/pg1‚0.0016320573546305178 & lr-AdamW/pg2‚0.0016320573546305178 trainer/global_step‚2449 % _timestamp‚1.7885387754206102e+09b2]ÝÌ¥Æ _step‚198 & lr-AdamW/pg1‚0.0016320573546305178 & lr-AdamW/pg2‚0.0016320573546305178 trainer/global_step‚2449 % _timestamp‚1.7885387754206102e+09 _runtime‚3570.731642262ÆZ˜ö$2¦- _wandbruntime‚3570  _runtime‚3570U‡²6ò'§ìO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14990987 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45762099 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30771110 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231104024613376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2809672119808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2799167175231 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206827473351168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206827473351168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2810582090752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113483240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2940589962752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚3570.733386481 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45761464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113456895 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209638073444864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228163702053376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62104127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14990354 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51352770 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228163678113280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚2449 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62104127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209637145470976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62130470 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202767797677039 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51352770 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231104292016128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206827491354112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202768722691399 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199968647541168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199968630501808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51352768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2809672119808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209637145470976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 267402752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144ÑgÌ ¨ñO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14990987 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45762099 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30771110 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚231104024613376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2809672119808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2799167175231 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206827473351168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206827473351168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2810582090752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113483240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2940589962752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _runtime‚3570.733386481 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45761464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113456895 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209638073444864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228163702053376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62104127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 GADeviceStatsMonitor.on_train_batch_start/oversize_seg¹Qs+ments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 243462656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14990354 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51352770 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228163678113280 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚2449 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62104127 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209637145470976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62130470 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202767797677039 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51352770 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚231104292016128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206827491354112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202768722691399 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199968647541168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199968630501808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51352768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2809672119808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209637145470976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 267402752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2810793580032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113501886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62155327 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200047648096352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51372928 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206909110439936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62128958 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202848857559429 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206909110439936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113528257 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206909092436992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2800284448509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113501886 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2811703564288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206909092436992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45779123 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202847932544861 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30783175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51372928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51372930 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231195668557312 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62155327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113528257 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2941740799488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51372930 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2801192423717 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209720814004224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14995948 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228253927757824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248öëÜ'«ûM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2811703564288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62128958 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45779757 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2941499447296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200047665135712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209720814004224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228253903817728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7885387767819514e+09 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2810793580032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113501886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62155327 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200047648096352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51372928 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206909110439936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62128958 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202848857559429 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206909110439936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113528257 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206909092436992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2800284448509 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 265292288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113501886 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2811703564288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206909092436992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45779123 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202847932544861 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30783175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51372928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51372930 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚231195668557312 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62155327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113528257 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2941740799488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51372930 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2801192423717 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209720814004224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14995948 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228253927757824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185ȃºd2¬- _wandbruntime‚3572  _runtime‚3572Ô7Xö[­Õ 0loss/FRt1/assignment_loss‚3.9432147753004934 1loss/SRqqt4/detection_loss‚1.0774861870436219 % _timestamp‚1.7885387767846777e+09 _runtime‚3572.095504935 0loss/FRt4/assignment_loss‚2.3857686983214483 2loss/SRqqt3/assignment_loss‚1.7486676667865955 0loss/SRqqt3/detection_loss‚3.253056460305264 .loss/FBt2/detection_loss‚1.084978643928965 0loss/FBt1/assignment_loss‚0.3740827359545689 1loss/SRqqt1/detection_loss‚2.5088743009874896 1loss/SRqqt2/detection_loss‚2.2418020580496107 /loss/FBt1/detection_loss‚2.7419304778942695 /loss/FBt3/detection_loss‚1.5129607727130254 trainer/global_step‚2449 2loss/SRqqt4/assignment_loss‚1.3107669657521595 .loss/FRt2/detection_loss‚2.882523393034935 .loss/FBt4/detection_loss‚5.069932460784912 0loss/FRt2/assignment_loss‚3.1807757115364073 0loss/FBt2/assignment_loss‚0.1914919255413121 /loss/FRt4/detection_loss‚1.7057030348183213 epoch‚2 _step‚201 0loss/FRt3/assignment_loss‚3.8987583458423614 /loss/FRt1/detection_loss‚2.2157727265182663 &loss/total_loss‚53.327296892240696 1loss/SRqqt1/assignment_loss‚1.806470213638198 0loss/FBt3/assignment_loss‚0.5588840462267394 2loss/SRqqt2/assignment_loss‚1.9632327764161996 0loss/FBt4/assignment_loss‚2.7528865337371826 /loss/FRt3/detection_loss‚2.9172759811083475=ìh`®Ú 2loss/SRqqt2/assignment_loss‚1.9632327764161996 0loss/FBt4/assignment_loss‚2.7528865337371826 /loss/FRt3/detection_loss‚2.9172759811083475 0loss/FRt1/assignment_loss‚3.9432147753004934 1loss/SRqqt4/detection_loss‚1.0774861870436219 % _timestamp‚1.7885387767846777e+09 _runtime‚3572.095504935 0loss/FRt4/assignment_loss‚2.3857686983214483 2loss/SRqqt3/assignment_loss‚1.7486676667865955 0loss/SRqqt3/detection_loss‚3.253056460305264 .loss/FBt2/detection_loss‚1.084978643928965 0loss/FBt1/assignment_loss‚0.3740827359545689 1loss/SRqqt1/detection_loss‚2.5088743009874896 1loss/SRqqt2/detection_loss‚2.2418020580496107 /loss/FBt1/detection_loss‚2.7419304778942695 /loss/FBt3/detection_loss‚1.5129607727130254 trainer/global_step‚2449 2loss/SRqqt4/assignment_loss‚1.3107669657521595 .loss/FRt2/detection_loss‚2.882523393034935 .loss/FBt4/detection_loss‚5.069932460784912 0loss/FRt2/assignment_loss‚3.1807757115364073 0loss/FBt2/assignment_loss‚0.1914919255413121 /loss/FRt4/detection_loss‚1.7057030348183213 epoch‚2 _step‚201 0loss/FRt3/assignment_loss‚3.8987583458423614 /loss/FRt1/detection_loss‚2.2157727265182663 &loss/total_loss‚53.327296892240696 1loss/SRqqt1/assignment_loss‚1.806470213638198 0loss/FBt3/assignment_loss‚0.5588840462267394É@W+“¯‚(:ˆ ¥ßëÔ½ðÄË gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ¢w°‚(:ì ¥ßëÔé—ÊÌ disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 122.5078125 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚71040.55615234375 disk.dm-1.in‚0 disk.dm-1.out‚ 122.5078125 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 8470.4609375( disk.nvme0c0n1.out‚71040.55615234375$ memory_percent‚91.94824695587158% disk./.usageGB‚1178.9466514587402 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 cpu‚25.420632123484133 proc.cpu.threads‚17 network.sent‚ 5.517631e+06 network.recv‚ 1.16464e+06 disk.sda3.out‚ 122.5078125 disk.dm-0.in‚ 8470.4609375 disk.drbd1260.out‚0.515625 disk.nvme0n1.in‚ 8470.4609375& disk.nvme0n1.out‚71040.55615234375) proc.memory.availableMB‚ 1319.19921875( disk./.usagePercent‚79.1200989815242šücž±‚(:“ ´ßëÔų Ê gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚148.975' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.794999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÞ½²‚(:ô ´ßëÔ–©¶Ì disk.sda.out‚ 122.83203125 disk.dm-0.in‚ 8527.19921875) proc.memory.availableMB‚ 1319.19140625 proc.memory.rssMB‚ 3073.53125 network.recv‚ 1.170844e+06# disk.nvme0c0n1.in‚ 8527.19921875( disk.nvme0c0n1.out‚71323.92919921875 disk.sda1.in‚0) disk./.usagePercent‚79.12009642144886 disk.drbd1260.out‚ 0.5234375! disk.nvme0n1.in‚ 8527.19921875 disk.sda.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.75934600830078 cpu‚25.419384672991942 proc.cpu.threads‚17% disk./.usageGB‚1178.9466133117676 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚71323.92919921875# disk.dm-0.out‚71323.92919921875# memory_percent‚91.9482946395874 network.sent‚ 5.626288e+06 disk.dm-1.out‚ 122.83203125 disk.sda3.in‚0 disk.sda3.out‚ 122.83203125ç‰Ûj³‚(:’ ÃßëÔÌøÌÊ gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.74' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.948000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýmÂéš´‚(:æ ÃßëÔó–ÛÍ disk.dm-0.in‚ 8628.1875' proc.memory.availableMB‚ 1319.453125 proc.memory.rssMB‚ 3073.53125 network.sent‚ 5.63351òÞ•×1e+06 disk.nvme0c0n1.in‚ 8628.1875 disk.nvme0n1.in‚ 8628.1875$ memory_percent‚91.94669723510742 proc.cpu.threads‚17% disk./.usageGB‚1178.9466514587402 disk.sda2.in‚0( disk.nvme0c0n1.out‚71472.47607421875 disk.sda1.in‚0# disk.dm-0.out‚71472.47607421875 disk.dm-1.in‚0 disk.dm-1.out‚ 123.62109375 cpu‚25.418058478173375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.5234375& disk.nvme0n1.out‚71472.47607421875 disk.sda3.out‚ 123.62109375 disk.sda.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.75934600830078 network.recv‚ 1.176902e+06( disk./.usagePercent‚79.1200989815242 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda.out‚ 123.62109375zÚ¼2µ- _wandbruntime‚3629  _runtime‚3629¢ÂƶÀ & lr-AdamW/pg2‚0.0016277120830129703 trainer/global_step‚2499 $ _timestamp‚1.788538834492547e+09 _runtime‚3629.803544048 _step‚202 & lr-AdamW/pg1‚0.0016277120830129703ôÄ‹ÿË·Å trainer/global_step‚2499 $ _timestamp‚1.788538834492547e+09 _runtime‚3629.803544048 _step‚202 & lr-AdamW/pg1‚0.0016277120830129703 & lr-AdamW/pg2‚0.0016277120830129703Êøò–Â2¸- _wandbruntime‚3629  _runtime‚3629í%Žñ'¹ëO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213775115161088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998673335808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2855044430339 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚210908445645312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235673636671488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2855952405339 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63349029 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213774187187200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865759544832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865759544832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚210908445645312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚210908427642368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115736142 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203919510229008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52360768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46651729 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788538834493612e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213775115161088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15277367 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235673893588480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2998906312704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206775479673707 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866669515776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63375372 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232674963335680ÔÞ»Óö'ºðO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31374360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 $ _timestamp‚1.788538834493612e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213775115161088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15277367 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235673893588480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2998906312704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206775479673707 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866669515776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63375372 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232674963335680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976896 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213775115161088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998673335808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2855044430339 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚210908445645312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235673636671488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2855952405339 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63349029 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213774187187200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865759544832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2865759544832 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚210908445645312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚210908427642368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115736142 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203919510229008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52360768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46651729 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63373860 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115781159 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52380928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2999849671168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206854689527169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚2499 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867790989312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46669478 _runtime‚3631.045971831 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115754788 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52380928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31386427 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210990064731136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46668867 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232765212980224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213857855720448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574976 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115754788 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52380928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31386427 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210990064731136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46668867 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232765212980224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213857855720448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63373860 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115781159 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52380928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2999849671168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206854689527169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚2499 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867790989312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46669478 _runtime‚3631.045971831ÌHjm 2À- _wandbruntime‚3631  _runtime‚3631-›ŸEÁ¿ 0loss/FRt2/assignment_loss‚2.3227961709101996 /loss/FRt3/assignment_loss‚3.771475895455009 /loss/FRt4/assignment_loss‚2.043863268747722 loss/FBt4/assignment_loss‚0 /loss/FRt3/detection_loss‚2.7632022751005074 /loss/FRt4/detection_loss‚1.4705762835921183 $ _timestamp‚1.788538835738443e+09 1loss/FBt2/assignment_loss‚0.21971848208707187 .loss/FRt1/detection_loss‚2.662471614565168 .loss/FBt1/detection_loss‚2.926253020763397 /loss/FBt2/detection_loss‚1.2783673831595566 _step‚205 epoch‚2 2loss/SRqqt4/assignment_loss‚1.2967294839423473 0loss/FBt3/assignment_loss‚0.6056054992710832 0loss/SRqqt2/detection_loss‚2.355619840323925 .loss/FBt3/detection_loss‚1.604675631312763 /loss/FBt4/detection_loss‚0.6563465595245361 %loss/total_loss‚47.54124702374831 trainer/global_step‚2499 1loss/SRqqt1/assignment_loss‚2.661988465558915 2loss/SRqqt3/assignment_loss‚1.8661093684760006 2loss/SRqqt2/assignment_loss‚2.1773792331417403 0loss/SRqqt1/detection_loss‚3.223277225380852 /loss/FRt1/assignment_loss‚4.679212760925293 0loss/FBt1/assignment_loss‚0.3570419242818995 1loss/SRqqt4/detection_loss‚1.0139824227085263 _runtime‚3631.049970129 1loss/SRqqt3/detection_loss‚2.9135606397281992 /loss/FRt2/detection_loss‚2.6709935747914844n¾þóJÂÄ 0loss/SRqqt1/detection_loss‚3.223277225380852 /loss/FRt1/assignment_loss‚4.679212760925293 0loss/FBt1/assignment_loss‚0.3570419242818995 1loss/SRqqt4/detection_loss‚1.0139824227085263 _runtime‚3631.049970129 1loss/SRqqt3/detection_loss‚2.9135606397281992 /loss/FRt2/detection_loss‚2.6709935747914844 0loss/FRt2/assignment_loss‚2.3227961709101996 /loss/FRt3/assignment_loss‚3.771475895455009 /loss/FRt4/assignment_loss‚2.043863268747722 loss/FBt4/assignment_loss‚0 /loss/FRt3/detection_loss‚2.7632022751005074 /loss/FRt4/detection_loss‚1.4705762835921183 $ _timestamp‚1.788538835738443e+09 1loss/FBt2/assignment_loss‚0.21971848208707187 .loss/FRt1/detection_loss‚2.662471614565168 .loss/FBt1/detection_loss‚2.926253020763397 /loss/FBt2/detection_loss‚1.2783673831595566 _step‚205 epoch‚2 2loss/SRqqt4/assignment_loss‚1.2967294839423473 0loss/FBt3/assignment_loss‚0.6056054992710832 0loss/SRqqt2/detection_loss‚2.355619840323925 .loss/FBt3/detection_loss‚1.604675631312763 /loss/FBt4/detection_loss‚0.6563465595245361 %loss/total_loss‚47.54124702374831 trainer/global_step‚2499 1loss/SRqqt1/assignment_loss‚2.661988465558915 2loss/SRqqt3/assignment_loss‚1.8661093684760006 2loss/SRqqt2/assignment_loss‚2.1773792331417403ͨP£÷“Â(:ˆ áßëÔÒÕ‰Ì gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^vrÄ‚(:ç áßëÔÆ”‰Î disk.nvme0n1.in‚ 8628.1875 disk.dm-0.in‚ 8628.1875 disk.dm-1.out‚ 124.63671875 disk.sda2.in‚0$ memory_percent‚91.94862842559814 proc.memory.rssMB‚ 3073.53125) disk./.usagePercent‚79.12011946212694% disk.nvme0n1.out‚72039.2861328125 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚25.415605116400553 network.sent‚ 5.738156e+06% disk./.usageGB‚1178.9469566345215 disk.sda3.in‚0 disk.sda3.out‚ 124.63671875 disk.sda.out‚ 124.63671875' disk.nvme0c0n1.out‚72039.2861328125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.52734375 disk.nvme0c0n1.in‚ 8628.1875) proc.memory.availableMB‚ 1319.13671875) proc.memory.percent‚18.75934600830078" disk.dm-0.out‚72039.2861328125 network.recv‚ 1.191729e+06|L“Å‚(:ˆ ðßëÔ„¼Ê gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.677' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04›qÆ‚(:æ ðßëÔ‰ìñË disk.drbd1260.in‚0) proc.memory.availableMB‚ 1319.38671875 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 cpu‚25.414292671948928 proc.cpu.threads‚17) disk./.usagePercent‚79.12011562201393 disk.sda2.out‚0 disk.drbd1260.out‚0.53125( disk.nvme0c0n1.out‚72380.32373046875 disk.sda3.out‚ 124.98828125 disk.sda.in‚0 disk.sda.out‚ 124.98828125 disk.dm-0.in‚ 8628.1875 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 8628.1875 disk.nvme0n1.in‚ 8628.1875& disk.nvme0n1.out‚72380.32373046875 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 124.98828125 network.sent‚ 5.837185e+06 network.recv‚ 1.198916e+06% disk./.usageGB‚1178.9468994140625 disk.sda1.in‚0# disk.dm-0.out‚72380.32373046875# memory_percent‚91.9471025466919¯§žÇ‚(:“ ÿßëÔéÜãÌ gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.786' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.357200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ ¢IqÈ‚(:æ ÿßëÔÒíøÎ$ memory_percent‚91.94834232330322) proc.memory.percent‚18.75934600830078 cpu‚25.41305011888808 network.sent‚ 5.856504e+06 disk.sda2.in‚0& disk.nvme0n1.out‚72663.46435546875 disk.sda3.in‚0) proc.memory.availableMB‚ 1319.18359375 proc.cpu.threads‚17) disk./.usagePercent‚79.12012407026255 disk.dm-1.out‚125.4375 disk.sda.in‚0 disk.sda.out‚125.4375 disk.sda1.in‚0 proc.memory.rssMB‚ 3073.53125 network.recv‚ 1.229573e+06# disk.dm-0.out‚72663.46435546875 disk.dm-1.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 8831.51953125( disk.nvme0c0n1.out‚72663.46435546875! disk.nvme0n1.in‚ 8831.51953125 disk.sda3.out‚125.4375% disk./.usageGB‚1178.9470252990723 disk.dm-0.in‚ 8831.51953125 disk.sda2.out‚0 disk.drbd1260.out‚0.53125 disk.sda1.out‚0›$Ê 2É- _wandbruntime‚3689  _runtime‚3689!4ÇYÅÊ¿ _step‚206 % lr-AdamW/pg1‚0.001623203225712706 % lr-AdamW/pg2‚0.001623203225712706 trainer/global_step‚2549 % _timestamp‚1.7885388940672925e+09 _runtime‚3689.378241112’Ž ?ÊËÄ trainer/global_step‚2549 % _timestamp‚1.7885388940672925e+09 _runtime‚3689.378241112 _step‚206 % lr-AdamW/pg1‚0.001623203225712706 % lr-AdamW/pg2‚0.001623203225712706νaÔ2Ì- _wandbruntime‚3689  _runtime‚3689@‘mò'ÍìO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207870389956208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240243527157760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64593931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31977612 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2922756940800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2911829660447 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2910921685447 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64620274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207870406995568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2921846969856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240243282823680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210781311641655 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31977610 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217911228903424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2922756940800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚214989399936512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237186272498176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217911228903424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237186248558080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210782236656015 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53368770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210782236656015 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53368770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207870389956208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240243527157760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64593931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31977612 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2922756940800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2911829660447 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2910921685447 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64620274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207870406995568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2921846969856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240243282823680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210781311641655 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31977610 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217911228903424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2922756940800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚214989399936512 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237186272498176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217911228903424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237186248558080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47559471 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53388930 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118007690 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237276498202624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53388930 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217993969449472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885388952112238e+09 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210861446509477 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚3690.522836764 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215071001019392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215071019022336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215071001019392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47560099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118034061 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15570422 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64645131 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2922968430080 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64618762 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31989677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31989675 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118034061 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15570422 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64645131 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2922968430080 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64618762 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31989677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31989675 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47559471 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53388930 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118007690 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚237276498202624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53388930 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217993969449472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885388952112238e+09 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210861446509477 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚3690.522836764 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215071001019392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237157888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215071019022336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215071001019392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47560099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222048270619648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48431070 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚3748.362323575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2978844365824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120215601 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120241946 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244812861841408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214788993638323 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244813114564096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219070336224768 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2978844365824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222048270619648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222049198593536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65865176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241697557720576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219070354227712 trainer/global_step‚2599¼¨÷'âñO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244813114564096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219070336224768 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2978844365824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222048270619648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222049198593536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65865176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241697557720576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219070354227712 trainer/global_step‚2599 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54376768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15850208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚628 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214788068623963 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54376770 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚211 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32580862 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2977934394880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65838833 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 % _timestamp‚1.7885389530511389e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219070336224768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2977934394880 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211821286722768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2966798940555 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15849580 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54376768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65865176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3115328060928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222048270619648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48431070 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚3748.362323575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2978844365824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120215601 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120241946 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244812861841408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214788993638323 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640Óoã¶2ã- _wandbruntime‚3749  _runtime‚3749Æ ·2û&äõM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2979965839360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222131939152896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54396930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54396928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 trainer/global_step‚2599 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120286963 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15855265 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219151955310592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65863664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214869128506353 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2968824189041 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3116485734912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65890033 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54396930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48448190 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32592927 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚3749.466127745 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54396928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2979055855104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241787759484928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120286963 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65890033 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15855897 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211900304317312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 _step‚212 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65863664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚219151955310592 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222131939152896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219151973313536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120260592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48448824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244904245219840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222131011165696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241787783425024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214868203491785 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788538954154546e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222131939152896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219151973313536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120260592 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48448824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244904245219840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222131011165696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241787783425024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214868203491785 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788538954154546e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120286963 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15855265 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219151955310592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65863664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214869128506353 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2968824189041 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3116485734912 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65890033 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54396930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48448190 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32592927 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚3749.466127745 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200Ôóz×2æ- _wandbruntime‚3749  _runtime‚3749 êqEWçÑ 2loss/SRqqt4/assignment_loss‚1.3347126254573083 .loss/FRt2/detection_loss‚2.191899344750813 /loss/FRt4/detection_loss‚1.5873049673857167 /loss/FRt1/assignment_loss‚3.340832527341514 0loss/FRt4/assignment_loss‚2.1795253830296653 0loss/SRqqt2/detection_loss‚2.626349382675611 _step‚213 2loss/SRqqt1/assignment_loss‚1.6798761947766732 2loss/SRqqt2/assignment_loss‚1.6249887158091252 /loss/FBt2/detection_loss‚0.8563161828179857 /loss/FBt3/detection_loss‚1.6810862863293061 0loss/FRt2/assignment_loss‚2.5018490621021816 1loss/SRqqt1/detection_loss‚2.1055651495331213 1loss/SRqqt4/detection_loss‚1.0391973856613677 .loss/FBt1/detection_loss‚2.952475334916796 &loss/total_loss‚51.956188898158004 epoch‚2 0loss/FBt3/assignment_loss‚0.5151487388167807 .loss/FRt1/detection_loss‚2.969725408430757 0loss/SRqqt3/detection_loss‚2.484361288448175 0loss/FRt3/assignment_loss‚3.6769199986611643 1loss/SRqqt3/assignment_loss‚2.054511702309052 0loss/FBt2/assignment_loss‚0.2901344271037021 /loss/FRt3/detection_loss‚2.9372497066374748 .loss/FBt4/detection_loss‚6.265861749649048 _runtime‚3749.467995386 0loss/FBt1/assignment_loss‚0.4153531445356618 /loss/FBt4/assignment_loss‚2.644944190979004 trainer/global_step‚2599 % _timestamp‚1.7885389541572132e+09Ìâ¯\èÖ 2loss/SRqqt1/assignment_loss‚1.6798761947766732 2loss/SRqqt2/assignment_loss‚1.6249887158091252 /loss/FBt2/detection_loss‚0.8563161828179857 /loss/FBt3/detection_loss‚1.6810862863293061 0loss/FRt2/assignment_loss‚2.5018490621021816 1loss/SRqqt1/detection_loss‚2.1055651495331213 1loss/SRqqt4/detection_loss‚1.0391973856613677 .loss/FBt1/detection_loss‚2.952475334916796 &loss/total_loss‚51.956188898158004 epoch‚2 0loss/FBt3/assignment_loss‚0.5151487388167807 .loss/FRt1/detection_loss‚2.969725408430757 0loss/SRqqt3/detection_loss‚2.484361288448175 0loss/FRt3/assignment_loss‚3.6769199986611643 1loss/SRqqt3/assignment_loss‚2.054511702309052 0loss/FBt2/assignment_loss‚0.2901344271037021 /loss/FRt3/detection_loss‚2.9372497066374748 .loss/FBt4/detection_loss‚6.265861749649048 _runtime‚3749.467995386 0loss/FBt1/assignment_loss‚0.4153531445356618 /loss/FBt4/assignment_loss‚2.644944190979004 trainer/global_step‚2599 % _timestamp‚1.7885389541572132e+09 2loss/SRqqt4/assignment_loss‚1.3347126254573083 .loss/FRt2/detection_loss‚2.191899344750813 /loss/FRt4/detection_loss‚1.5873049673857167 /loss/FRt1/assignment_loss‚3.340832527341514 0loss/FRt4/assignment_loss‚2.1795253830296653 0loss/SRqqt2/detection_loss‚2.626349382675611 _step‚213ÕÆÓFwé‚(:ì ÊàëÔˆ‡éÎ% disk.nvme0n1.out‚73991.7451171875$ memory_percent‚96.26986980438232) disk./.usagePercent‚79.13156530297529% disk./.usageGB‚1179.1175079345703 disk.sda1.out‚0" disk.dm-0.out‚73991.7451171875 disk.drbd1260.out‚0.546875' disk.nvme0c0n1.out‚73991.7451171875( proc.memory.availableMB‚ 611.14453125 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 127.83984375 disk.sda1.in‚0 disk.dm-0.in‚ 8954.1953125 disk.dm-1.out‚ 127.83984375 disk.drbd1260.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 8954.1953125 disk.sda3.out‚ 127.83984375 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 cpu‚25.40726536376924 proc.cpu.threads‚17 disk.sda2.in‚0 network.sent‚ 6.089317e+06 network.recv‚ 1.272236e+06 disk.dm-1.in‚0 disk.nvme0n1.in‚ 8954.1953125Þ9k–“ê‚(:ˆ ÊàëÔéÿÇÏ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'­H8“ë‚(:ˆ ÙàëÔ’ÄûË gpu.0.gpu‚26.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.707' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E™èGwì‚(:ì ÙàëÔð¼«Í proc.cpu.threads‚17 network.sent‚ 6.186081e+06% disk./.usageGB‚1179.1170616149902 disk.dm-1.out‚ 128.24609375 disk.sda2.out‚0' disk.nvme0c0n1.out‚74423.2060546875% disk.nvme0n1.out‚74423.2060546875 disk.sda3.in‚0 disk.sda3.out‚ 128.24609375 network.recv‚ 1.277944e+06) disk./.usagePercent‚79.13153535009378 disk.dm-0.in‚ 8954.1953125" disk.dm-0.out‚74423.2060546875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.546875" disk.nvme0c0n1.in‚ 8954.1953125 disk.sda.out‚ 128.24609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 8954.1953125 disk.sda.in‚0$ memory_percent‚96.27382755279541) proc.memory.percent‚18.75934600830078( proc.memory.availableMB‚ 610.49609375 proc.memory.rssMB‚ 3073.53125 cpu‚25.40620838853089z³ës“í‚(:ˆ èàëÔþÊ gpu.0.gpu‚42.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ï6krî‚(:ç èàëÔƒÆ·Í disk.sda2.in‚0 disk.drbd1260.out‚ 0.55078125) proc.memory.percent‚18.75934600830078 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 128.5703125$ disk./.usageGB‚1179.179443359375 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 8954.1953125$ disk.nvme0n1.out‚74486.298828125 disk.sda1.in‚0 disk.dm-0.in‚ 8954.1953125 network.sent‚ 6.190125e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 128.5703125! disk.dm-0.out‚74486.298828125 disk.dm-1.in‚0$ memory_percent‚96.27394676208496' proc.memory.availableMB‚ 610.4765625 network.recv‚ 1.282738e+06) disk./.usagePercent‚79.13572184130122& disk.nvme0c0n1.out‚74486.298828125 disk.nvme0n1.in‚ 8954.1953125 disk.sda1.out‚0 disk.dm-1.out‚ 128.5703125 proc.memory.rssMB‚ 3073.53125 cpu‚25.404942092851876^tg“ï‚(:ˆ ÷àëÔ„ø‡Í gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!§?hð‚(:Ý ÷àëÔù†ÅÐ) proc.memory.percent‚18.75934600830078! disk.nvme0c0n1.in‚ 9083.234375 disk.sda.out‚129.3125 disk.dm-1.out‚129.3125 proc.cpu.threads‚17 network.recv‚ 1.287284e+06) disk./.usagePercent‚79.13834515050458 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55078125 disk.sda3.in‚0( proc.memory.availableMB‚ 610.20703125 cpu‚25.403904474136116 network.sent‚ 6.196311e+06% disk./.usageGB‚1179.2185325622559 disk.sda2.in‚0& disk.nvme0c0n1.out‚74932.185546875 disk.nvme0n1.in‚ 9083.234375$ disk.nvme0n1.out‚74932.185546875 disk.sda3.out‚129.3125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 9083.234375! disk.dm-0.out‚74932.185546875 disk.dm-1.in‚0$ memory_percent‚96.27559185028076 proc.memory.rssMB‚ 3073.53125·ù›ô2ñ- _wandbruntime‚3809  _runtime‚3809sœ´úÅò¿ & lr-AdamW/pg1‚0.0016136985465156327 & lr-AdamW/pg2‚0.0016136985465156327 trainer/global_step‚2649 $ _timestamp‚1.788539014336225e+09 _runtime‚ 3809.64722336 _step‚214ëýVôÊóÄ _runtime‚ 3809.64722336 _step‚214 & lr-AdamW/pg1‚0.0016136985465156327 & lr-AdamW/pg2‚0.0016136985465156327 trainer/global_step‚2649 $ _timestamp‚1.788539014336225e+09Ö+`0+2ô- _wandbruntime‚3809  _runtime‚3809}6}ð'õêO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49320872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3023584170663 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226186240309760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246208819002880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218795750620631 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215772166449968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122494848 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16137377 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16136762 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚215 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 3809.64891047 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226186240309760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226185312335872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3174046947328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55384770 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55384770 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223151290515968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246208842942976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218794825606271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚249382889890304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3173813970432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33184112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249382632973312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223151308518912 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67083735 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3022676195663 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223151308518912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3034021819904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788539014337736e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3034021819904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152$ ’ÁtöïO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33184112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249382632973312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223151308518912 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67083735 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3022676195663 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223151308518912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3034021819904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 $ _timestamp‚1.788539014337736e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3034021819904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49320872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3023584170663 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226186240309760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246208819002880 JDDevicÄì&}eStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218795750620631 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215772166449968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122494848 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16137377 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16136762 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚215 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 3809.64891047 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226186240309760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226185312335872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3174046947328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55384770 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55384770 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223151290515968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246208842942976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218794825606271 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚249382889890304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3173813970432×—C “÷‚(:ˆ †áëÔëÛçË gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{1S&tø‚(:é †áëÔÑ–âÍ disk.sda3.out‚ 129.9921875$ memory_percent‚96.27988338470459 proc.cpu.threads‚17 network.recv‚ 1.29071e+06 disk.dm-1.out‚ 129.9921875 disk.sda2.in‚0( disk.nvme0c0n1.out‚75278.06396484375 disk.sda.in‚0 disk.sda.out‚ 129.9921875 network.sent‚ 6.201268e+06) disk./.usagePercent‚79.14676959043382% disk./.usageGB‚1179.3440628051758 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.5546875 disk.sda3.in‚0) proc.memory.percent‚18.75934600830078 disk.dm-0.in‚ 9083.234375# disk.dm-0.out‚75278.06396484375 disk.drbd1260.in‚0! disk.nvme0c0n1.in‚ 9083.234375& disk.nvme0n1.out‚75278.06396484375( proc.memory.availableMB‚ 609.50390625 proc.memory.rssMB‚ 3073.53125 cpu‚25.40282585290626 disk.sda1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 9083.234375&sÝR2ù- _wandbruntime‚3810  _runtime‚3810µÅ(Sü&úöM _step‚216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3035143280128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 trainer/global_step‚2649 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49338684 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67108566 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚249474019641856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55404930 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3036053264384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55404928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16142509 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7885390155190232e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223232927604736 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3036053264384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246299044707328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3175207898112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚249474276545536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215851184044512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55404928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33196175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215851167005152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223232909601792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122539865 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226268980869120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3023793468941 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122539865 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3035143280128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223232909601792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚226268052881920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246299068647424 _runtime‚3810.831687956 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226268980869120 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3036053264384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246299044707328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3175207898112 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚249474276545536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚215851184044512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55404928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33196175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215851167005152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223232909601792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122539865 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226268980869120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3023793468941 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122539865 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3035143280128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223232909601792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚226268052881920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246299068647424 _runtime‚3810.831687956 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226268980869120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49338684 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67108566 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚249474019641856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55404930 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3036053264384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55404928غêv2ü- _wandbruntime‚3810  _runtime‚3810ÒÎçÙDý¾ 2loss/SRqqt4/assignment_loss‚1.3045773176880655 /loss/SRqqt1/detection_loss‚1.98508436350446 /loss/FBt4/detection_loss‚0.7711029052734375 &loss/total_loss‚48.643648340404624 1loss/SRqqt1/assignment_loss‚1.660986693257368 /loss/FRt3/detection_loss‚2.7674327194690704 /loss/FBt1/detection_loss‚2.3741235303878785 epoch‚2 /loss/FRt3/assignment_loss‚4.618608403205871 1loss/SRqqt2/assignment_loss‚2.329845376809438 .loss/FRt4/detection_loss‚1.849930759554616 % _timestamp‚1.7885390155229146e+09 0loss/FBt1/assignment_loss‚0.5125498510792386 loss/FBt4/assignment_loss‚0 0loss/SRqqt2/detection_loss‚3.287508753935496 /loss/FBt3/detection_loss‚2.0042541947215797 /loss/FRt1/assignment_loss‚4.106659244088566 0loss/FRt4/assignment_loss‚2.1533738483082163 0loss/FBt3/assignment_loss‚0.4315595164398232 .loss/FBt2/detection_loss‚1.397534918849883 2loss/SRqqt3/assignment_loss‚2.3844144323776506 /loss/FRt1/detection_loss‚2.7501166620675255 1loss/SRqqt4/detection_loss‚1.0306576407810515 trainer/global_step‚2649 _runtime‚3810.833744623 0loss/FRt2/assignment_loss‚3.4989139086160903 .loss/FRt2/detection_loss‚2.178253177648936 0loss/SRqqt3/detection_loss‚3.091522899167291 _step‚217 0loss/FBt2/assignment_loss‚0.1546372231730748öq»(Iþà 2loss/SRqqt3/assignment_loss‚2.3844144323776506 /loss/FRt1/detection_loss‚2.7501166620675255 1loss/SRqqt4/detection_loss‚1.0306576407810515 trainer/global_step‚2649 _runtime‚3810.833744623 0loss/FRt2/assignment_loss‚3.4989139086160903 .loss/FRt2/detection_loss‚2.178253177648936 0loss/SRqqt3/detection_loss‚3.091522899167291 _step‚217 0loss/FBt2/assignment_loss‚0.1546372231730748 2loss/SRqqt4/assignment_loss‚1.3045773176880655 /loss/SRqqt1/detection_loss‚1.98508436350446 /loss/FBt4/detection_loss‚0.7711029052734375 &loss/total_loss‚48.643648340404624 1loss/SRqqt1/assignment_loss‚1.660986693257368 /loss/FRt3/detection_loss‚2.7674327194690704 /loss/FBt1/detection_loss‚2.3741235303878785 epoch‚2 /loss/FRt3/assignment_loss‚4.618608403205871 1loss/SRqqt2/assignment_loss‚2.329845376809438 .loss/FRt4/detection_loss‚1.849930759554616 % _timestamp‚1.7885390155229146e+09 0loss/FBt1/assignment_loss‚0.5125498510792386 loss/FBt4/assignment_loss‚0 0loss/SRqqt2/detection_loss‚3.287508753935496 /loss/FBt3/detection_loss‚2.0042541947215797 /loss/FRt1/assignment_loss‚4.106659244088566 0loss/FRt4/assignment_loss‚2.1533738483082163 0loss/FBt3/assignment_loss‚0.4315595164398232 .loss/FBt2/detection_loss‚1.397534918849883Ù k6“ÿ‚(:ˆ •áëÔõ—áÍ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚176.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gζu€‚(:ê •áëÔñ°«Î disk.drbd1260.out‚ 0.5546875( disk.nvme0c0n1.out‚75638.87646484375& disk.nvme0n1.out‚75638.87646484375 disk.sda.in‚0 disk.sda.out‚ 130.48046875# disk.dm-0.out‚75638.87646484375 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0c0n1.in‚ 9083.234375 disk.nvme0n1.in‚ 9083.234375 disk.dm-1.in‚0$ memory_percent‚96.27418518066406) proc.memory.percent‚18.75934600830078% disk./.usageGB‚1179.3674926757812 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 130.48046875$ proc.memory.availableMB‚610.4375 proc.memory.rssMB‚ 3073.53125 cpu‚25.401774221517677 network.recv‚ 1.296231e+06 disk.sda3.in‚0 disk.sda3.out‚ 130.48046875 disk.dm-0.in‚ 9083.234375 proc.cpu.threads‚17 network.sent‚ 6.297828e+06) disk./.usagePercent‚79.14834198870912†Æ—;ž‚(:“ ¤áëÔüˆ‰Ì gpu.0.gpu‚18.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚118.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.799400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Ñ 0x‚‚(:í ¤áëÔã†èÏ cpu‚25.40065481102497 network.recv‚ 1.302303e+06 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 9083.234375 disk.sda.out‚ 131.00390625' proc.memory.availableMB‚ 609.8515625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.out‚ 0.55859375( disk.nvme0c0n1.out‚75940.68994140625 disk.nvme0n1.in‚ 9083.234375 disk.sda3.out‚ 131.00390625# disk.dm-0.out‚75940.68994140625$ memory_percent‚96.27776145935059 network.sent‚ 6.395717e+06) disk./.usagePercent‚79.16518933252242 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 9083.234375) proc.memory.percent‚18.75934600830078% disk./.usageGB‚1179.6185302734375 disk.drbd1260.in‚0& disk.nvme0n1.out‚75940.68994140625 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 131.00390625 proc.memory.rssMB‚ 3073.53125œæ¼xƒ‚(:í ³áëÔù×ÖÌ disk.sda2.out‚0 proc.memory.rssMB‚ 3073.53125 cpu‚25.39960563012201 disk.sda3.out‚ 131.66796875 network.sent‚ 6.40393e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55859375! disk.nvme0c0n1.in‚ 9199.046875 disk.dm-0.in‚ 9199.046875 disk.sda2.in‚0) proc.memory.percent‚18.75934600830078 network.recv‚ 1.307248e+06) disk./.usagePercent‚79.16512353858612 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 610.06640625 proc.cpu.threads‚17% disk./.usageGB‚1179.6175498962402( disk.nvme0c0n1.out‚76171.65478515625 disk.nvme0n1.in‚ 9199.046875& disk.nvme0n1.out‚76171.65478515625# disk.dm-0.out‚76171.65478515625 disk.dm-1.out‚ 131.66796875$ memory_percent‚96.27645015716553 disk.sda.out‚ 131.66796875 disk.dm-1.in‚0~¯Ð]‹„‚(:ˆ ³áëÔ©Í gpu.0.gpu‚49.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedByt«¬Àfes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚135.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.1718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UcD2…- _wandbruntime‚3869  _runtime‚3869ÕƆÀ _step‚218 & lr-AdamW/pg1‚0.0016087047053797217 & lr-AdamW/pg2‚0.0016087047053797217 trainer/global_step‚2699 $ _timestamp‚1.788539074266928e+09 _runtime‚3869.578004393TóƒË‡Å _runtime‚3869.578004393 _step‚218 & lr-AdamW/pg1‚0.0016087047053797217 & lr-AdamW/pg2‚0.0016087047053797217 trainer/global_step‚2699 $ _timestamp‚1.788539074266928e+09Ú×ì/:2ˆ- _wandbruntime‚3869  _runtime‚3869,A'“ñ'‰ëO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124721405 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50213175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230322354052096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885390742686229e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3091019215872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3078553450771 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3091019215872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56392768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253952505180160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222801582588579 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _step‚219 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227232262810112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33787360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3232135649280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68328637 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227232244807168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219723029137808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚227232262810112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227232244807168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68354980 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16426450 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16425815 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230322354052096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265305600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885390742686229e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3091019215872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3078553450771 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3091019215872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56392768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253952505180160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222801582588579 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 _step‚219 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227232262810112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33787360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3232135649280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68328637 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227232244807168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219723029137808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚227232262810112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚227232244807168Ûš]¡u‘‹‚(:† ÂáëÔÒ¢õË gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.87' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.774 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d”wpŒ‚(:å ÂáëÔ›˜ÕÐ( disk.nvme0c0n1.out‚76473.76123046875$ memory_percent‚96.27959728240967 network.recv‚ 1.310723e+06 disk.dm-0.in‚ 9253.21875 disk.sda3.out‚ 132.2109375 disk.sda.out‚ 132.2109375( proc.memory.availableMB‚ 609.55078125 proc.memory.rssMB‚ 3073.53125 network.sent‚ 6.407092e+06 disk.dm-1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 9253.21875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.75934600830078 cpu‚25.398468533303202% disk./.usageGB‚1179.6175689697266 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.5625& disk.nvme0n1.out‚76473.76123046875 disk.sda3.in‚0 disk.sda.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚79.16512481862378# disk.dm-0.out‚76473.76123046875 disk.dm-1.out‚ 132.2109375 disk.nvme0c0n1.in‚ 9253.21875Ä’%â2- _wandbruntime‚3870  _runtime‚3870ñžžJû&ŽõM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227313881895936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227313863892992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219802063771712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3080578699257 _runtime‚3870.965089508 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230406022585344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3092140689408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50231463 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254043888590336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56412928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3091230705152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124792767 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56412930 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3092140689408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124766396 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250810329929728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219802046732352 _step‚220 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 $ _timestamp‚1.788539075652964e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68353468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16432036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3091230705152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33799425 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250810353869824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3233534720512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56412930 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227313881895936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124766396 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68379837 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50230818 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68353468 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222881717456401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124792767 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227313863892992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230406022585344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230405094598144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079670724049 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56412928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3091230705152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124792767 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56412930 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230866432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3092140689408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124766396 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250810329929728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219802046732352 _step‚220 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 $ _timestamp‚1.788539075652964e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68353468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16432036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3091230705152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33799425 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250810353869824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3233534720512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56412930 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚227313881895936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124766396 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68379837 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50230818 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68353468 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222881717456401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124792767 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚227313863892992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230406022585344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230405094598144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079670724049 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517Ü;ºÖ2- _wandbruntime‚3870  _runtime‚3870G žX‘Ò epoch‚2 $ _timestamp‚1.788539075656715e+09 2loss/SRqqt3/assignment_loss‚2.3625302029152713 /loss/FBt4/detection_loss‚5.8933712641398115 %loss/total_loss‚53.47770244688652 0loss/FRt1/assignment_loss‚3.8412236744357693 2loss/SRqqt2/assignment_loss‚1.6412799897102208 0loss/FBt3/assignment_loss‚0.4219861859335861 0loss/FRt4/assignment_loss‚2.1492888541288777 0loss/FBt4/assignment_loss‚2.6439860661824546 0loss/SRqqt1/detection_loss‚2.143289368403585 0loss/FRt2/assignment_loss‚2.8014779119956783 /loss/FBt1/assignment_loss‚0.436426948655439 /loss/FRt1/detection_loss‚2.8796293648981277 /loss/FRt2/detection_loss‚2.3472603450461134 1loss/SRqqt4/detection_loss‚0.9823773786059169 _step‚221 /loss/FRt3/assignment_loss‚4.218069558342298 /loss/FRt3/detection_loss‚2.9658202131589255 1loss/SRqqt2/detection_loss‚2.3398097386726966 /loss/FBt1/detection_loss‚2.3384097584269266 /loss/FBt3/detection_loss‚2.2006425349151386 _runtime‚3870.967685823 1loss/SRqqt1/assignment_loss‚1.978097339994029 0loss/SRqqt3/detection_loss‚2.709883642693361 trainer/global_step‚2699 1loss/FBt2/assignment_loss‚0.14668889845269695 .loss/FBt2/detection_loss‚1.169075783525334 1loss/SRqqt4/assignment_loss‚1.234958193249634 /loss/FRt4/detection_loss‚1.6321192304046275èªï¾]’× /loss/FRt2/detection_loss‚2.3472603450461134 1loss/SRqqt4/detection_loss‚0.9823773786059169 _step‚221 /loss/FRt3/assignment_loss‚4.218069558342298 /loss/FRt3/detection_loss‚2.9658202131589255 1loss/SRqqt2/detection_loss‚2.3398097386726966 /loss/FBt1/detection_loss‚2.3384097584269266 /loss/FBt3/detection_loss‚2.2006425349151386 _runtime‚3870.967685823 1loss/SRqqt1/assignment_loss‚1.978097339994029 0loss/SRqqt3/detection_loss‚2.709883642693361 trainer/global_step‚2699 1loss/FBt2/assignment_loss‚0.14668889845269695 .loss/FBt2/detection_loss‚1.169075783525334 1loss/SRqqt4/assignment_loss‚1.234958193249634 /loss/FRt4/detection_loss‚1.6321192304046275 epoch‚2 $ _timestamp‚1.788539075656715e+09 2loss/SRqqt3/assignment_loss‚2.3625302029152713 /loss/FBt4/detection_loss‚5.8933712641398115 %loss/total_loss‚53.47770244688652 0loss/FRt1/assignment_loss‚3.8412236744357693 2loss/SRqqt2/assignment_loss‚1.6412799897102208 0loss/FBt3/assignment_loss‚0.4219861859335861 0loss/FRt4/assignment_loss‚2.1492888541288777 0loss/FBt4/assignment_loss‚2.6439860661824546 0loss/SRqqt1/detection_loss‚2.143289368403585 0loss/FRt2/assignment_loss‚2.8014779119956783 /loss/FBt1/assignment_loss‚0.436426948655439 /loss/FRt1/detection_loss‚2.8796293648981277Ýî?kq““‚(:ˆ ÑáëÔÀª†Ë gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:¿m”‚(:â ÑáëÔ¶’©Í disk.sda3.out‚ 132.5078125 disk.sda.in‚0# disk.dm-0.out‚76733.84619140625 disk.dm-1.in‚0 disk.sda2.out‚0 network.recv‚ 1.316184e+06 disk.nvme0c0n1.in‚ 9253.21875& disk.nvme0n1.out‚76733.84619140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 disk.sda.out‚ 132.5078125 disk.dm-0.in‚ 9253.21875 disk.drbd1260.out‚0.5625( proc.memory.availableMB‚ 609.74609375 proc.cpu.threads‚17( disk.nvme0c0n1.out‚76733.84619140625 disk.nvme0n1.in‚ 9253.21875 disk.sda3.in‚0 disk.dm-1.out‚ 132.5078125 disk.sda2.in‚0$ memory_percent‚96.27840518951416 cpu‚25.39743882104736 network.sent‚ 6.503737e+06( disk./.usagePercent‚79.1654404759135$ disk./.usageGB‚1179.622272491455v4•‚(:’ àáëÔ“ÕË gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.33' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.666000000000004 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bks–‚(:è àáëÔʳíÎ network.sent‚ 6.601659e+06) disk./.usagePercent‚79.16386167744984$ disk./.usageGB‚1179.598747253418 disk.drbd1260.out‚ 0.56640625& disk.nvme0n1.out‚77228.31591796875# disk.dm-0.out‚77228.31591796875 disk.dm-1.out‚ 132.8828125$ memory_percent‚96.27845287322998 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 9253.21875( disk.nvme0c0n1.out‚77228.31591796875 disk.sda3.in‚0 disk.sda3.out‚ 132.8828125 disk.sda.out‚ 132.8828125 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 disk.sda2.in‚0 disk.nvme0n1.in‚ 9253.21875 disk.sda1.out‚0 disk.dm-0.in‚ 9253.21875 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.322281e+06 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 609.73828125 cpu‚25.396541858833537yÌò“—‚(:ˆ ïáëÔù›•Ì gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î’«Ðt˜‚(:é ïáëÔÐû«Î$ disk./.usageGB‚1180.122272491455 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 133.55859375 disk.sda.in‚0 disk.sda.out‚ 133.55859375 network.sent‚ 6.607992e+06( disk.nvme0c0n1.out‚77483.27685546875& disk.nvme0n1.out‚77483.27685546875 disk.dm-0.in‚ 9253.21875 disk.dm-1.out‚ 133.55859375& proc.memory.availableMB‚ 609.734375 proc.memory.rssMB‚ 3073.53125 cpu‚25.395489207983722 disk.drbd1260.out‚ 0.56640625 disk.nvme0c0n1.in‚ 9253.21875 disk.dm-1.in‚0$ memory_percent‚96.27847671508789) proc.memory.percent‚18.75934600830078 network.recv‚ 1.326845e+06 disk.nvme0n1.in‚ 9253.21875 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚77483.27685546875 proc.cpu.threads‚17) disk./.usagePercent‚79.19899589544144âê9ž™‚(:“ þáëÔ—ܘÎ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.755' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.550999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é°üpš‚(:å þáëÔÖúžÑ disk.sda3.out‚ 134.0859375 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚96.27845287322998 proc.memory.rssMB‚ 3073.53125 network.sent‚ 6.612918e+06$ disk.nvme0n1.out‚77766.763671875 disk.sda3.in‚0 disk.sda.out‚ 134.0859375 disk.dm-0.in‚ 9380.0078125 disk.dm-1.out‚ 134.0859375 disk.sda2.out‚0 disk.drbd1260.out‚ 0.5703125% disk./.usageGB‚1180.1222915649414 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 609.73828125 cpu‚25.3945259438188 network.recv‚ 1.33034e+06& disk.nvme0c0n1.out‚77766.763671875 disk.sda.in‚0! disk.dm-0.out‚77766.763671875) proc.memory.percent‚18.75934600830078 proc.cpu.threads‚17) disk./.usagePercent‚79.19899717547911" disk.nvme0c0n1.in‚ 9380.0078125 disk.nvme0n1.in‚ 9380.0078125Éd·2›- _wandbruntime‚3931  _runtime‚3931ºý‡Åœ¿ % lr-AdamW/pg1‚0.001603551239670104 % lr-AdamW/pg2‚0.001603551239670104 trainer/global_step‚2749 % _timestamp‚1.7885391363689165e+09 _runtime‚3931.679880192 _step‚222ø² ÊÄ _runtime‚3931.679880192 _step‚222 % lr-AdamW/pg1‚0.001603551239670104 % lr-AdamW/pg2‚0.001603551239670104 trainer/global_step‚2749 % _timestamp‚1.7885391363689165e+09ÞUýB2ž- _wandbruntime‚3931  _runtime‚3931…õò'ŸìO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234459395768320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51104014 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226808339570887 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69573539 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234459395768320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51104636 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231313199098368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57400768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69599882 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226809264585247 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚223673925904368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57400770 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69573539 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3147106640896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3931.681181685 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57400770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69599882 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34390612 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231313199098368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885391363700743e+09·ØÒ˜+ ñO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69573539 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚234459395768320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51104636 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231313199098368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57400768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69599882 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625536 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226809264585247 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚223673925904368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57400770 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69573539 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3147106640896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3931.681181685 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57400770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69599882 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34390612 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚231313199098368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885391363700743e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚234459395768320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51104014 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226808339570887 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ß/€½2¡- _wandbruntime‚3932  _runtime‚3932Ú·²šü&¢öM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16719709 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16719099 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255321615152128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127019298 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3148228114432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51122386 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51121774 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3135547979157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚231394836187136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885391374236753e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57420930 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226888474438709 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34402675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127045669 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34402677 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223752943498912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255321639092224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223752926459552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57420928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69624739 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231394818184192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3291966178816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127045669 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3147318130176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127019298 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 _step‚224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69598370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3136455954365 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3148228114432 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3147318130176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚234543064301568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231394836187136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234542136314368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258613605271040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚3932.735267752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚234543064301568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69624739 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258613363047424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3291747895296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57420928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226889399453277‚QºF'£ûM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 trainer/global_step‚2749 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57420930 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231394818184192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57420930 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226888474438709 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34402675 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127045669 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34402677 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚223752943498912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255321639092224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223752926459552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57420928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69624739 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚231394818184192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3291966178816 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127045669 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3147318130176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127019298 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 _step‚224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69598370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3136455954365 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3148228114432 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3147318130176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚234543064301568 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231394836187136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚234542136314368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚258613605271040 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚3932.735267752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚234543064301568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69624739 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258613363047424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3291747895296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57420928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226889399453277 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16719709 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16719099 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255321615152128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283520 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127019298 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3148228114432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51122386 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51121774 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3135547979157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚231394836187136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 % _timestamp‚1.7885391374236753e+09 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596437484544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238597365458432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3203194065920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70818441 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34993862 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3203194065920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3202284094976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3190307960987 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3202284094976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58408768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259742674670080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235394171392512 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259742698610176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58408768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263091858967040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58408770 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230815096553195 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615wmIö'´ðO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3202284094976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3190307960987 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3202284094976 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58408768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚259742674670080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235394171392512 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259742698610176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58408768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263091858967040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58408770 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230815096553195 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70844784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17001797 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3191215935987 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129227209 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129253554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17002418 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70844784 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚3992.388838224 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230816021567555 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚2799 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51995657 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596437484544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238597365458432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3203194065920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70818441 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34993862 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3203194065920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144㥮u\2µ- _wandbruntime‚3993  _runtime‚3993%¡“‹û&¶õM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58428930 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3204315539456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238679178030592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35005927 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚228 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17007451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58428928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70843272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52013376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263183249334272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129298571 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230896156435585 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259832900374528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230895231421017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235475772475392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3350106736128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235475790478336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3204315539456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3203405555200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129272200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58428928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238680106017792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788539198189033e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238680106017792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58428930 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227703806186752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70843272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235475790478336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35005925 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3191425234265 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚3993.501157967 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52013987 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3204315539456 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238679178030592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35005927 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _step‚228 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17007451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242223616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58428928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70843272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52013376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263183249334272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129298571 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚230896156435585 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259832900374528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230895231421017 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235475772475392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3350106736128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235475790478336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3204315539456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3203405555200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129272200 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58428928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238680106017792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788539198189033e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238680106017792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58428930 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚227703806186752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70843272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚235475790478336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35005925 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3191425234265 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 _runtime‚3993.501157967 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52013987 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856ä'Z9"2¸- _wandbruntime‚3993  _runtime‚3993A3á A¹» /loss/FRt2/assignment_loss‚2.351222478982174 0loss/FRt4/assignment_loss‚1.9940553865855253 2loss/SRqqt1/assignment_loss‚2.1513935116029557 2loss/SRqqt3/assignment_loss‚2.1966762904609953 2loss/SRqqt4/assignment_loss‚1.4702845221304375 1loss/SRqqt4/detection_loss‚1.1040890380416228 1loss/FBt2/assignment_loss‚0.27341276112203716 /loss/FBt2/detection_loss‚1.0091524283955062 % _timestamp‚1.7885391981920717e+09 /loss/FRt3/detection_loss‚2.7197874015708092 'loss/FBt4/detection_loss‚ 0.95703125 0loss/FBt3/assignment_loss‚0.5445619106523261 loss/FBt4/assignment_loss‚0 /loss/FRt2/detection_loss‚2.0919875330997235 1loss/SRqqt3/detection_loss‚2.8021590688398907 trainer/global_step‚2799 /loss/FRt1/detection_loss‚2.5237419463512376 /loss/FBt1/detection_loss‚2.5179706532508135 epoch‚2 _step‚229 1loss/SRqqt2/assignment_loss‚2.071852801022706 1loss/SRqqt2/detection_loss‚2.3628760631437653 /loss/FRt3/assignment_loss‚4.482021146350437 0loss/FBt1/assignment_loss‚0.5104513500627945 .loss/FRt4/detection_loss‚1.676741039828409 /loss/FBt3/detection_loss‚2.1361226615580646 /loss/FRt1/assignment_loss‚4.199497106598645 /loss/SRqqt1/detection_loss‚2.67993890277801 &loss/total_loss‚46.827027252428884 _runtime‚3993.502858052­F¾_FºÀ /loss/FRt1/detection_loss‚2.5237419463512376 /loss/FBt1/detection_loss‚2.5179706532508135 epoch‚2 _step‚229 1loss/SRqqt2/assignment_loss‚2.071852801022706 1loss/SRqqt2/detection_loss‚2.3628760631437653 /loss/FRt3/assignment_loss‚4.482021146350437 0loss/FBt1/assignment_loss‚0.5104513500627945 .loss/FRt4/detection_loss‚1.676741039828409 /loss/FBt3/detection_loss‚2.1361226615580646 /loss/FRt1/assignment_loss‚4.199497106598645 /loss/SRqqt1/detection_loss‚2.67993890277801 &loss/total_loss‚46.827027252428884 _runtime‚3993.502858052 /loss/FRt2/assignment_loss‚2.351222478982174 0loss/FRt4/assignment_loss‚1.9940553865855253 2loss/SRqqt1/assignment_loss‚2.1513935116029557 2loss/SRqqt3/assignment_loss‚2.1966762904609953 2loss/SRqqt4/assignment_loss‚1.4702845221304375 1loss/SRqqt4/detection_loss‚1.1040890380416228 1loss/FBt2/assignment_loss‚0.27341276112203716 /loss/FBt2/detection_loss‚1.0091524283955062 % _timestamp‚1.7885391981920717e+09 /loss/FRt3/detection_loss‚2.7197874015708092 'loss/FBt4/detection_loss‚ 0.95703125 0loss/FBt3/assignment_loss‚0.5445619106523261 loss/FBt4/assignment_loss‚0 /loss/FRt2/detection_loss‚2.0919875330997235 1loss/SRqqt3/detection_loss‚2.8021590688398907 trainer/global_step‚2799å÷Ë?Jœ»‚(:‘ ÉâëÔÞüêÏ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚152.9' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.580000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o½f¼‚(:Û ÉâëÔ¡µëÐ% disk.nvme0c0n1.out‚79238.67578125 disk.sda1.in‚0 disk.dm-0.out‚79238.67578125 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 network.sent‚ 6.742155e+06 disk.nvme0n1.in‚ 9519.4609375# disk.nvme0n1.out‚79238.67578125 disk.sda.in‚0 disk.dm-0.in‚ 9519.4609375# memory_percent‚96.2775707244873 cpu‚25.389574339338935 disk.drbd1260.out‚0.578125" disk.nvme0c0n1.in‚ 9519.4609375 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0( disk./.usagePercent‚79.1728178450277% disk./.usageGB‚1179.7322006225586 disk.sda2.out‚0 disk.sda3.out‚ 136.28125 disk.sda.out‚ 136.28125 disk.dm-1.out‚ 136.28125' proc.memory.availableMB‚ 609.8828125 proc.cpu.threads‚17 network.recv‚ 1.351695e+06 disk.sda2.in‚0 disk.drbd1260.in‚0òKƒ“½‚(:ˆ ØâëÔÖÅÒÌ gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.927' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÁžq¾‚(:æ ØâëÔ¥œ×Ñ) proc.memory.percent‚18.75934600830078 cpu‚25.388557202245227 disk.sda.in‚0 disk.dm-1.out‚ 136.984375' proc.memory.availableMB‚ 609.3828125 proc.cpu.threads‚17 network.recv‚ 1.358046e+06) disk./.usagePercent‚79.17281912506537' disk.nvme0c0n1.out‚79670.5771484375 disk.nvme0n1.in‚ 9519.4609375 disk.sda3.in‚0 disk.dm-0.in‚ 9519.4609375" disk.dm-0.out‚79670.5771484375 disk.sda2.out‚0 network.sent‚ 6.850923e+06$ disk./.usageGB‚1179.732219696045 disk.sda.out‚ 136.984375 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.58203125# memory_percent‚96.2806224822998 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 9519.4609375% disk.nvme0n1.out‚79670.5771484375 disk.sda3.out‚ 136.984375 disk.dm-1.in‚0 proc.memory.rssMB‚ 3073.53125wI"Ç“¿‚(:ˆ çâëÔ‘ÿêÉ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.079' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.6158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚˆïrÀ‚(:ç çâëÔʯ³Ì network.recv‚ 1.3631e+06) disk./.usagePercent‚79.17279378031948" disk.nvme0c0n1.in‚ 9519.4609375 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0) proc.memory.percent‚18.75934600830078 disk.nvme0n1.in‚ 9519.4609375% disk.nvme0n1.out‚79829.6005859375 disk.sda.out‚ 137.453125 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.58203125( proc.memory.availableMB‚ 610.19921875 network.sent‚ 6.859242e+06% disk./.usageGB‚1179.7318420410156' disk.nvme0c0n1.out‚79829.6005859375 disk.sda3.out‚ 137.453125 disk.dm-0.in‚ 9519.4609375 disk.dm-1.out‚ 137.453125$ memory_percent‚96.27563953399658 proc.memory.rssMB‚ 3073.53125 disk.sda1.in‚0" disk.dm-0.out‚79829.6005859375 cpu‚25.387565370273823 proc.cpu.threads‚17L˜Ì“Á‚(:ˆ öâëÔÖæîË gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Û¡z‚(:ï öâëÔÄÿ¬Ñ network.sent‚ 6.862397e+06 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 9708.3515625( disk.nvme0c0n1.out‚79983.43896484375 cpu‚25.386631859927355 proc.cpu.threads‚17$ disk./.usageGB‚1179.731861114502# disk.dm-0.out‚79983.43896484375 disk.dm-1.out‚ 138.24609375 disk.nvme0n1.in‚ 9708.3515625 disk.sda3.in‚0 disk.sda.in‚0 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 disk.dm-0.in‚ 9708.3515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.5859375& disk.nvme0n1.out‚79983.43896484375 disk.sda.out‚ 138.24609375 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.366652e+06) disk./.usagePercent‚79.17279506035716 disk.dm-1.in‚0 disk.sda3.out‚ 138.24609375$ memory_percent‚96.27566337585449' proc.memory.availableMB‚ 610.1953125%Âðe2Ã- _wandbruntime‚4053  _runtime‚4053$ ÇÄÁ & lr-AdamW/pg1‚0.0015927697634695742 & lr-AdamW/pg2‚0.0015927697634695742 trainer/global_step‚2849 % _timestamp‚1.7885392584595366e+09 _runtime‚4053.770557022 _step‚230ýÕÌÅÆ & lr-AdamW/pg1‚0.0015927697634695742 & lr-AdamW/pg2‚0.0015927697634695742 trainer/global_step‚2849 % _timestamp‚1.7885392584595366e+09 _runtime‚4053.770557022 _step‚230æ¹[[º2Æ- _wandbruntime‚4053  _runtime‚4053HRiò'ÇìO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3259281490944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247093191095 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239475107680768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35597112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267661306641408 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17290816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131480111 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59416770 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52887305 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264253959892480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52887928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35597110 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239475107680768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242733479200768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242733479200768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231575668319408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3407102414848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234822778549863 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267661062307328 trainer/global_step‚2849 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131506456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72089686 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7885392584606905e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59416768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242734407174656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264253983832576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239475125683712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72063343 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3407322808832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242734407174656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59416768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72063343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231575685358768 _runtime‚4053.771827427 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239475125683712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3258371520000®+.ô)ÈñO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234822778549863 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267661062307328 trainer/global_step‚2849 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131506456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72089686 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7885392584606905e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _step‚231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59416768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242734407174656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264253983832576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚239475125683712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72063343 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3407322808832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242734407174656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59416768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72063343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23157568535âªæÓÎ8768 _runtime‚4053.771827427 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚239475125683712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3258371520000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3259281490944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3247093191095 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚239475107680768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35597112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267661306641408 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17290816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131480111 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59416770 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52887305 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264253959892480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52887928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35597110 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚239475107680768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242733479200768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242733479200768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231575668319408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3407102414848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0çlÄö~2É- _wandbruntime‚4054  _runtime‚4054‘„§¸ü&ÊöM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242816219746816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131525102 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267752701352448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚232 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59436928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3408491815424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59436930 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259492980224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35609175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72088174 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234902913417893 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131551473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72114543 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239556726766592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 247643648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234901988403325 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231654685913952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131551473 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267752429768704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35609177 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17295905 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264344209537024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131525102 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239556726766592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52905080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248210464581 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52905698 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231654702953312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚239556744769536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59436928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚4054.768230624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517^ô*'ËûM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131525102 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚239556726766592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52905080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3248210464581 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52905698 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231654702953312 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚239556744769536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59436928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚4054.768230624 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242816219746816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131525102 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267752701352448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚232 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59436928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3408491815424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59436930 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259492980224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35609175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72088174 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234902913417893 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131551473 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72114543 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚239556726766592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 247643648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234901988403325 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231654685913952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131551473 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267752429768704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35609177 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17295905 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264344209537024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389èð]]¡2Ì- _wandbruntime‚4054  _runtime‚4054¥7 TÍÎ /loss/FBt4/assignment_loss‚2.426731586456299 .loss/FRt3/detection_loss‚2.513780406189846 /loss/FBt2/detection_loss‚1.1225382182747126 .loss/FBt3/detection_loss‚1.817968153084318 % _timestamp‚1.7885392594593077e+09 _runtime‚4054.770089308 0loss/FRt2/assignment_loss‚2.4998177736997604 1loss/SRqqt3/assignment_loss‚2.046717369556427 /loss/FRt1/detection_loss‚2.8187424847573945 1loss/SRqqt4/detection_loss‚1.0074639250713635 .loss/FBt4/detection_loss‚6.735869646072388 trainer/global_step‚2849 2loss/SRqqt4/assignment_loss‚1.2665489313109377 0loss/SRqqt2/detection_loss‚2.241786717340864 /loss/FRt1/assignment_loss‚3.434145703460231 1loss/SRqqt2/assignment_loss‚1.676411230759374 1loss/SRqqt3/detection_loss‚2.9198827177286146 /loss/FBt1/detection_loss‚2.8313607082647434 epoch‚2 1loss/FBt2/assignment_loss‚0.29067324121972904 0loss/FBt3/assignment_loss‚0.5755830657047518 .loss/FRt2/detection_loss‚2.056605210900307 %loss/total_loss‚55.53299354995892 0loss/SRqqt1/detection_loss‚3.350636218275343 1loss/SRqqt1/assignment_loss‚2.898413430367197 0loss/FBt1/assignment_loss‚0.5195678043014863 /loss/FRt4/detection_loss‚1.9061376715048415 /loss/FRt3/assignment_loss‚4.124108334382375 0loss/FRt4/assignment_loss‚2.4515030012756096 _step‚233èÄö@YÎÓ 1loss/SRqqt1/assignment_loss‚2.898413430367197 0loss/FBt1/assignment_loss‚0.5195678043014863 /loss/FRt4/detection_loss‚1.9061376715048415 /loss/FRt3/assignment_loss‚4.124108334382375 0loss/FRt4/assignment_loss‚2.4515030012756096 _step‚233 /loss/FBt4/assignment_loss‚2.426731586456299 .loss/FRt3/detection_loss‚2.513780406189846 /loss/FBt2/detection_loss‚1.1225382182747126 .loss/FBt3/detection_loss‚1.817968153084318 % _timestamp‚1.7885392594593077e+09 _runtime‚4054.770089308 0loss/FRt2/assignment_loss‚2.4998177736997604 1loss/SRqqt3/assignment_loss‚2.046717369556427 /loss/FRt1/detection_loss‚2.8187424847573945 1loss/SRqqt4/detection_loss‚1.0074639250713635 .loss/FBt4/detection_loss‚6.735869646072388 trainer/global_step‚2849 2loss/SRqqt4/assignment_loss‚1.2665489313109377 0loss/SRqqt2/detection_loss‚2.241786717340864 /loss/FRt1/assignment_loss‚3.434145703460231 1loss/SRqqt2/assignment_loss‚1.676411230759374 1loss/SRqqt3/detection_loss‚2.9198827177286146 /loss/FBt1/detection_loss‚2.8313607082647434 epoch‚2 1loss/FBt2/assignment_loss‚0.29067324121972904 0loss/FBt3/assignment_loss‚0.5755830657047518 .loss/FRt2/detection_loss‚2.056605210900307 %loss/total_loss‚55.53299354995892 0loss/SRqqt1/detection_loss‚3.350636218275343é A1˜žÏ‚(:“ …ãëÔÒã¬Ì gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.184' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.436799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 4è}xЂ(:í …ãëÔãÀãÐ disk.sda.in‚0 disk.sda1.in‚0# disk.dm-0.out‚80338.71630859375$ memory_percent‚96.27740383148193) proc.memory.percent‚18.75934600830078 network.sent‚ 6.867592e+06 network.recv‚ 1.370358e+06% disk./.usageGB‚1179.7093505859375 disk.drbd1260.in‚0 disk.sda3.out‚ 138.5703125 disk.dm-1.out‚ 138.5703125 cpu‚25.38571588882198 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 9708.3515625( disk.nvme0c0n1.out‚80338.71630859375& disk.nvme0n1.out‚80338.71630859375 disk.sda1.out‚0 disk.dm-0.in‚ 9708.3515625 proc.memory.rssMB‚ 3073.53125 proc.cpu.threads‚17) disk./.usagePercent‚79.17128435989761 disk.sda2.out‚0 disk.nvme0n1.in‚ 9708.3515625 disk.sda.out‚ 138.5703125 disk.dm-1.in‚0( proc.memory.availableMB‚ 609.91015625 disk.drbd1260.out‚ 0.5859375 disk.sda3.in‚0/ªÌçžÑ‚(:“ ”ãëÔÝé‡Î gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚133.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.679799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄG/sÒ‚(:è ”ãëÔì·òÎ disk.dm-1.out‚ 139.015625 disk.drbd1260.out‚ 0.58984375( proc.memory.availableMB‚ 610.14453125 proc.memory.rssMB‚ 3073.53125 proc.cpu.threads‚17) disk./.usagePercent‚79.17285906224072 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 9708.3515625 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 9708.3515625' disk.nvme0c0n1.out‚80660.5439453125 disk.sda3.out‚ 139.015625 disk.sda.out‚ 139.015625" disk.dm-0.out‚80660.5439453125 disk.sda2.in‚0 disk.nvme0n1.in‚ 9708.3515625) proc.memory.percent‚18.75934600830078 cpu‚25.38482339997116 network.recv‚ 1.376513e+06 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚80660.5439453125$ memory_percent‚96.27597332000732 network.sent‚ 6.976125e+06% disk./.usageGB‚1179.7328147888184 disk.sda.in‚0q_7›“Ó‚(:ˆ £ãëÔ›†Í gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Þ§xÔ‚(:í £ãëÔÝæ¹Ñ' proc.memory.availableMB‚ 610.1484375 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 cpu‚25.383940536373398 proc.cpu.threads‚17% disk./.usageGB‚1179.7319259643555% disk.nvme0n1.out‚81198.4501953125 disk.sda.in‚0 disk.dm-0.in‚ 9708.3515625 disk.dm-1.out‚ 139.55078125' disk.nvme0c0n1.out‚81198.4501953125 network.sent‚ 6.982324e+06 network.recv‚ 1.38128e+06 disk.nvme0n1.in‚ 9708.3515625 disk.sda1.in‚0" disk.dm-0.out‚81198.4501953125 disk.sda2.in‚0$ memory_percent‚96.27594947814941 disk.sda3.out‚ 139.55078125 disk.sda.out‚ 139.55078125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.58984375" disk.nvme0c0n1.in‚ 9708.3515625) disk./.usagePercent‚79.17279941248523 disk.sda3.in‚0Ñr6ÅÕ‚(:ˆ ²ãëÔÙÇõË gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ ZŠ?14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚170.754' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'HkÑwÖ‚(:ì ²ãëÔ·©Ð cpu‚25.38299357913779 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚0.59375' disk.nvme0c0n1.out‚81467.4130859375! disk.nvme0n1.in‚ 9853.55078125% disk.nvme0n1.out‚81467.4130859375 disk.sda3.in‚0 proc.memory.rssMB‚ 3073.53125( disk./.usagePercent‚79.1728006925229 disk.sda1.in‚0 disk.dm-0.in‚ 9853.55078125" disk.dm-0.out‚81467.4130859375 disk.sda3.out‚ 140.43359375 disk.sda.in‚0$ memory_percent‚96.27594947814941) proc.memory.percent‚18.75934600830078 disk.dm-1.in‚0 disk.dm-1.out‚ 140.43359375 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 9853.55078125' proc.memory.availableMB‚ 610.1484375 network.sent‚ 6.987338e+06 network.recv‚ 1.384665e+06% disk./.usageGB‚1179.7319450378418 disk.drbd1260.in‚0 disk.sda.out‚ 140.43359375_ƒ 2×- _wandbruntime‚4115  _runtime‚4115 ÙC=ÅØ¿ _runtime‚4115.696339902 _step‚234 % lr-AdamW/pg1‚0.001587143999822142 % lr-AdamW/pg2‚0.001587143999822142 trainer/global_step‚2899 % _timestamp‚1.7885393203850942e+09ùSFWÊÙÄ _step‚234 % lr-AdamW/pg1‚0.001587143999822142 % lr-AdamW/pg2‚0.001587143999822142 trainer/global_step‚2899 % _timestamp‚1.7885393203850942e+09 _runtime‚4115.696339902êý«ÏÞ2Ú- _wandbruntime‚4115  _runtime‚41156`ñ'ÛëO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60424770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268765269054976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3302970446203 _runtime‚4115.697981938 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3314458945024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238829535532171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36200362 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243556079974912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53780427 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246870520916992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272231050260992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3465781206016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60424770 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243556061971968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246871448890880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133759358 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚235 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243556079974912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73334588 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580702 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185<°Ü£ö'ÜðO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17580067 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚268765245114880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73308245 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚243556079974912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53780427 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246870520916992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272231050260992 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3465781206016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60424770 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243556061971968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246871448890880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133759358 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 _step‚235 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243556079974912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73334588 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17580702 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60424770 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268765269054976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3302970446203 _runtime‚4115.697981938 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3314458945024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚238829535532171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36200362 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4117.075606501 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238909670400201 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60444930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36212425 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73359445 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133804375 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73333076 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3316490389504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804375 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243637681057792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17586500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133778004 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243637681057792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60444928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3466949942272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73333076 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53798304 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53798927 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268855494759424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3316490389504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36212427 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚2899 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3315580405248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3315580405248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246953261463040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17585879x4ª&2ßúM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53798304 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53798927 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268855494759424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3316490389504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36212427 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚2899 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3315580405248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3315580405248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246953261463040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17585879 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73359445 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4117.075606501 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238909670400201 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60444930 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36212425 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73359445 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133804375 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73333076 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3316490389504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804375 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243637681057792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17586500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133778004 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243637681057792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60444928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3466949942272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73333076 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3371456340992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276800860962816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239477444813168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74553147 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚2949 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61432770 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 % _timestamp‚1.7885393812216084e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61432770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3524077902848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61432768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61432768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247637034266112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3370546370048 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74579490 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247637016263168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251007562633216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36803612 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17869520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3371456340992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247637016263168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273276530337280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3371456340992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276800860962816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239477444813168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74553147 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚2949 PCDeviceStatsMonitor.on_traino Ò,_batch_start/active.large_pool.allocated‚61432770 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 % _timestamp‚1.7885393812216084e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61432770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3524077902848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61432768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61432768 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚247637034266112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3370546370048 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74579490 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚247637016263168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚251007562633216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36803612 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17869520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3371456340992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚247637016263168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚273276530337280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3371667830272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36815677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17875394 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61452930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74577978 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3525248454656 trainer/global_step‚2949 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136057277 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3359964974797 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239556462407712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74604347 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136030906 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273366756041728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136057277 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247718653351936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3372577814528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247718635348992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3525475126784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251091231166464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74604347 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54691069 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54691697 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251090303179264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251091231166464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242915502367941 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3371667830272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136030906 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61452930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74577978 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3525248454656 trainer/global_step‚2949 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136057277 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3359964974797 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239556462407712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74604347 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136030906 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚273366756041728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136057277 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚247718653351936 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3372577814528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚247718635348992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3525475126784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226672128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61452928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚251091231166464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74604347 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54691069 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54691697 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚251090303179264 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚251091231166464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚242915502367941 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3371667830272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136030906 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3371667830272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36815677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17875394 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ðP#³„2ô- _wandbruntime‚4177  _runtime‚4177¬‘YÜIõà .loss/FBt4/detection_loss‚1.112562656402588 _step‚241 0loss/FRt2/assignment_loss‚2.6403778238994318 2loss/SRqqt2/assignment_loss‚2.5783729975111784 2loss/SRqqt4/assignment_loss‚1.0117839491745877 .loss/FRt2/detection_loss‚2.217214700652332 /loss/FRt1/assignment_loss‚4.118598123391469 0loss/SRqqt2/detection_loss‚3.002921760082245 /loss/FBt1/detection_loss‚2.3019743164380393 /loss/FBt3/detection_loss‚1.5772744285372586 0loss/FRt3/assignment_loss‚3.2421342217259936 2loss/SRqqt1/assignment_loss‚1.5736981952643914 0loss/FBt3/assignment_loss‚0.6742125818362592 /loss/FRt3/detection_loss‚2.6109802401802478 1loss/SRqqt3/detection_loss‚2.5665087699890137 % _timestamp‚1.7885393822594066e+09 0loss/FRt4/assignment_loss‚1.5623333705589175 epoch‚2 loss/FBt4/assignment_loss‚0 /loss/FRt1/detection_loss‚2.8020622010032334 1loss/SRqqt1/detection_loss‚2.4901898632878843 /loss/FBt2/detection_loss‚1.0078692815665689 2loss/SRqqt3/assignment_loss‚1.8715693945134129 trainer/global_step‚2949 _runtime‚4177.570166244 0loss/FBt2/assignment_loss‚0.6047593201303617 1loss/SRqqt4/detection_loss‚1.0143992955526036 %loss/total_loss‚44.77302649644075 0loss/FBt1/assignment_loss‚0.5081253804770919 .loss/FRt4/detection_loss‚1.683103624265641†œÝNöÈ 1loss/SRqqt4/detection_loss‚1.0143992955526036 %loss/total_loss‚44.77302649644075 0loss/FBt1/assignment_loss‚0.5081253804770919 .loss/FRt4/detection_loss‚1.683103624265641 .loss/FBt4/detection_loss‚1.112562656402588 _step‚241 0loss/FRt2/assignment_loss‚2.6403778238994318 2loss/SRqqt2/assignment_loss‚2.5783729975111784 2loss/SRqqt4/assignment_loss‚1.0117839491745877 .loss/FRt2/detection_loss‚2.217214700652332 /loss/FRt1/assignment_loss‚4.118598123391469 0loss/SRqqt2/detection_loss‚3.002921760082245 /loss/FBt1/detection_loss‚2.3019743164380393 /loss/FBt3/detection_loss‚1.5772744285372586 0loss/FRt3/assignment_loss‚3.2421342217259936 2loss/SRqqt1/assignment_loss‚1.5736981952643914 0loss/FBt3/assignment_loss‚0.6742125818362592 /loss/FRt3/detection_loss‚2.6109802401802478 1loss/SRqqt3/detection_loss‚2.5665087699890137 % _timestamp‚1.7885393822594066e+09 0loss/FRt4/assignment_loss‚1.5623333705589175 epoch‚2 loss/FBt4/assignment_loss‚0 /loss/FRt1/detection_loss‚2.8020622010032334 1loss/SRqqt1/detection_loss‚2.4901898632878843 /loss/FBt2/detection_loss‚1.0078692815665689 2loss/SRqqt3/assignment_loss‚1.8715693945134129 trainer/global_step‚2949 _runtime‚4177.570166244 0loss/FBt2/assignment_loss‚0.6047593201303617ñ¸|‚ž÷‚(:“ ýãëÔæîÞÓ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.181' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.236200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j Æ©zø‚(:ï ýãëÔ®ÈÒÔ disk.sda1.out‚0 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078) disk./.usagePercent‚79.17286290235373% disk./.usageGB‚1179.7328720092773 disk.drbd1260.out‚ 0.6015625$ disk.nvme0c0n1.in‚10011.71484375& disk.nvme0n1.out‚83059.20458984375 disk.sda.in‚0 cpu‚25.37845620584118 disk.sda2.out‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚83059.20458984375" disk.nvme0n1.in‚10011.71484375 disk.sda3.in‚0 disk.dm-0.in‚10011.71484375# disk.dm-0.out‚83059.20458984375 disk.dm-1.out‚ 143.171875 network.recv‚ 1.406613e+06 disk.sda2.in‚0 disk.sda3.out‚ 143.171875 disk.sda.out‚ 143.171875 disk.dm-1.in‚0$ memory_percent‚96.27816677093506( proc.memory.availableMB‚ 609.78515625 proc.cpu.threads‚17 network.sent‚ 7.11761e+06 disk.sda1.in‚0gÚ/%žù‚(:“ ŒäëÔªÁ‘Î gpu.0.gpu‚15.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚157.102' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.420399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ2©ý~ú‚(:ó ŒäëÔ혜Ð( disk.nvme0c0n1.out‚83248.85302734375& disk.nvme0n1.out‚83248.85302734375 disk.sda.out‚ 143.83203125 disk.dm-0.in‚10011.71484375 disk.dm-1.in‚0 disk.dm-1.out‚ 143.83203125) proc.memory.percent‚18.75934600830078) disk./.usagePercent‚79.17285906224072% disk./.usageGB‚1179.7328147888184$ disk.nvme0c0n1.in‚10011.71484375# disk.dm-0.out‚83248.85302734375 proc.cpu.threads‚17 network.sent‚ 7.226126e+06 disk.drbd1260.in‚0" disk.nvme0n1.in‚10011.71484375 disk.sda3.in‚0 disk.sda3.out‚ 143.83203125 disk.sda.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 3073.53125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.60546875 disk.sda1.out‚0$ memory_percent‚96.27971649169922% proc.memory.availableMB‚ 609.53125 cpu‚25.3776449029474 network.recv‚ 1.412925e+06˜_·žû‚(:“ ›äëÔ„¬ÛÍ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.487' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.097399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒUqü‚(:æ ›äëÔɉºÐ! disk.nvme0c0n1.in‚ 10011.71875 disk.sda.in‚0 disk.sda.out‚ 144.421875 disk.sda1.out‚0$ memory_percent‚96.27666473388672% proc.memory.availableMB‚ 610.03125 proc.memory.rssMB‚ 3073.53125 disk.nvme0n1.in‚ 10011.71875 disk.sda3.out‚ 144.421875 disk.dm-1.out‚ 144.421875 cpu‚25.376723036564446 proc.cpu.threads‚17 disk.sda3.in‚0 network.recv‚ 1.418065e+06% disk./.usageGB‚1179.7094039916992 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.60546875& disk.nvme0n1.out‚83493.59521484375 disk.sda1.in‚0 network.sent‚ 7.232438e+06 disk.sda2.out‚0( disk.nvme0c0n1.out‚83493.59521484375 disk.dm-0.in‚ 10011.71875# disk.dm-0.out‚83493.59521484375 disk.dm-1.in‚0) proc.memory.percent‚18.75934600830078) disk./.usagePercent‚79.17128794400308êüf7ý‚(:† ªäëÔÒ¤ŽË gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.582 gpu.0.smClock‚1410 gpu.0.rý‰ÆZmemoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚmÄöwþ‚(:ì ªäëÔãÒúË disk.nvme0n1.in‚ 10011.71875 disk.sda3.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 609.78515625) proc.memory.percent‚18.75934600830078! disk.nvme0c0n1.in‚ 10011.71875 disk.sda1.out‚0# disk.dm-0.out‚83830.44189453125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚0.609375 proc.cpu.threads‚17 disk.sda.in‚0 network.sent‚ 7.239547e+06 network.recv‚ 1.421481e+06( disk.nvme0c0n1.out‚83831.31689453125 disk.sda3.out‚ 144.89453125 disk.drbd1260.in‚0& disk.nvme0n1.out‚83831.31689453125 disk.sda1.in‚0 disk.dm-0.in‚ 10011.71875 disk.dm-1.out‚ 144.89453125$ memory_percent‚96.27816677093506 proc.memory.rssMB‚ 3073.53125 cpu‚25.37592492230621 disk.sda.out‚ 144.89453125) disk./.usagePercent‚79.17127693567912% disk./.usageGB‚1179.7092399597168œëf"2ÿ- _wandbruntime‚4237  _runtime‚4237®Å€¿ % lr-AdamW/pg1‚0.001575428283192178 % lr-AdamW/pg2‚0.001575428283192178 trainer/global_step‚2999 % _timestamp‚1.7885394417888029e+09 _runtime‚4237.099783081 _step‚242à`ÍÊÄ % lr-AdamW/pg1‚0.001575428283192178 % lr-AdamW/pg2‚0.001575428283192178 trainer/global_step‚2999 % _timestamp‚1.7885394417888029e+09 _runtime‚4237.099783081 _step‚242òû€â‘2‚- _wandbruntime‚4237  _runtime‚4237µì<Ðò'ƒìO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277787815559680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18160574 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263208448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3427543766016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3582671810560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚243 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3427543766016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243428324540368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255144604349440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚251717988557312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37406862 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3413816981419 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37406860 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138238817 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281370750578688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55566794 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243428307501008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚251717970554368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255145532323328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75798049 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281370487370240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246843049496787 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251717988557312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3426633795072 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255145532323328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3426633795072Ù ¡ú÷'„ñO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62440768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚251717970554368 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277787839499776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75798049 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55567436 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243428307501008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚251717970554368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255145532323328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75798049 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281370487370240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246843049496787 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251717988557312 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3426633795072 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255145532323328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3426633795072 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277787815559680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18160574 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263208448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3427543766016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3582671810560 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚243 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3427543766016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243428324540368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255144604349440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚251717988557312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37406862 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3413816981419 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37406860 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138238817 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281370750578688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55566794 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62460930 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62460930 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75849249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3427755255296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255227344895488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3428665239552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75849249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55584624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62460928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251799607643136 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3427755255296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75822880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281462133730816 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18166340 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18165699 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚244 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75822880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281461881021440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138310179 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255228272882688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640é)ãj؇ûM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3427755255296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75822880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚281462133730816 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18166340 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18165699 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 _step‚244 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75822880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281461881021440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138310179 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255228272882688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255228272882688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277878041264128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62460930 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62460930 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75849249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3427755255296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255227344895488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3428665239552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75849249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55584624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62460928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251799607643136 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259281646065664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641298806784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63448768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18449690 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255798924845568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63448770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77069294 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56460421 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56459800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140491719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3483631191040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259282574039552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247379204267568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285940185486336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250848881464735 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3482721220096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259282574039552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282299100782080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250849806479095 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3469694236527 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚247379187228208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282299124722176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63448768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 % _timestamp‚1.7885395032391279e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63448770 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255798942848512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259281646065664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚247379187228208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚282299124722176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63448768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 % _timestamp‚1.7885395032391279e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63448770 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255798942848512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259281646065664 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259281646065664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641298806784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63448768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18449690 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255798924845568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63448770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77069294 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56460421 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56459800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140491719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3483631191040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259282574039552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247379204267568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285940185486336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250848881464735 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3482721220096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259282574039552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282299100782080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250849806479095 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3469694236527 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1÷ Àº'2™- _wandbruntime‚4299  _runtime‚4299Rz%òü&šöM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚255880561934336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56477669 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18456130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3483842680320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38022175 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885395044070568e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255880543931392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247458221862112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63468928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77094151 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚4299.719730174 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140536710 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3471719485013 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚255880543931392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18455494 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247458204822752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77067782 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259364386611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286031579422208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3470811509805 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282389350426624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140563081 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚250929016332557 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 trainer/global_step‚3049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38022177 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63468928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140536710 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3642481704960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255880561934336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259365314598912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282389326486528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642252935680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63468930 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56478307 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63468930 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140563081 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3471719485013 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚255880543931392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18455494 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247458204822752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77067782 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259364386611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286031579422208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3470811509805 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282389350426624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140563081 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚250929016332557 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 trainer/global_step‚3049 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38022177 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63468928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140536710 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3642481704960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚255880561934336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259365314598912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282389326486528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3642252935680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63468930 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56478307 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63468930 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140563081 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56477669 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18456130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3483842680320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38022175 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 % _timestamp‚1.7885395044070568e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255880543931392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247458221862112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63468928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77094151 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _runtime‚4299.719730174 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140536710 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259879897139712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3525571491635 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286810386004480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142744621 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229654016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254856563461403 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78314196 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286810409944576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259879879136768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38613362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚263418687781888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259879879136768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254855638447043 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251330083994768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64456768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263418687781888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263419615755776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539718616064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78314196 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57354028 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142770966 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64456770 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3538808645120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290510244362240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263419615755776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3538808645120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78287853 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18740666 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57353419 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3699858357760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64456768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290510474016256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3700064071680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚3099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78287853 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7885395623502796e+09 _runtime‚4357.661447562 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251330066955408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38613360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18740059 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539718616064˜ j÷'¬ñO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7885395623502796e+09 _runtime‚4357.661447562 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251330066955408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38613360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18740059 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539718616064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259879897139712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3525571491635 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286810386004480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142744621 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229654016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254856563461403 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚251 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78314196 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286810409944576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259879879136768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38613362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚263418687781888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259879879136768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254855638447043 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251330083994768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64456768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263418687781888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚263419615755776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539718616064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78314196 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57354028 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142770966 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64456770 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3538808645120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290510244362240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚263419615755776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3538808645120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78287853 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18740666 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57353419 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3699858357760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64456768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290510474016256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3700064071680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚3099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78287853 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615û¢SÌš2­- _wandbruntime‚4358  _runtime‚4358ý/Hü&®öM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254936698329433 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142815983 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚4358.792539143 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3540840089600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3526688764913 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259961516225536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38625427 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3527596740121 trainer/global_step‚3099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142789612 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885395634808855e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233835008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209894912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263502356315136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3539930105344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18745794 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540840089600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18746390 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263501428327936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64476928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3539930105344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142815983 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57371817 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259961516225536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57371219 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64476930 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290601664211456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64476930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64476928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251409084549952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78339053 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263502356315136 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3701262397440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 Ü6±¯ûM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3527596740121 trainer/global_step‚3099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142789612 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885395634808855e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233835008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209894912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263502356315136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3539930105344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18745794 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540840089600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18746390 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263501428327936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64476928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3539930105344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142815983 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57371817 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚259961516225536 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57371219 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64476930 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚290601664211456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64476930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64476928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251409084549952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78339053 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263502356315136 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3701262397440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254936698329433 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142815983 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _runtime‚4358.792539143 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3540840089600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3526688764913 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259961516225536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38625427 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144ü]{?|2°- _wandbruntime‚4358  _runtime‚4358ûdZV±Ð .loss/FRt1/detection_loss‚2.342750992701978 /loss/FRt3/detection_loss‚2.3649775106174755 /loss/FRt4/detection_loss‚1.8630534540861845 %loss/total_loss‚54.29086084410688 1loss/FBt2/assignment_loss‚0.20118533609981856 1loss/SRqqt1/detection_loss‚2.4743239264935255 .loss/FBt4/detection_loss‚7.022137641906738 1loss/SRqqt3/assignment_loss‚2.546648622705386 0loss/FRt4/assignment_loss‚1.5553013256617956 1loss/SRqqt1/assignment_loss‚2.089624732732773 2loss/SRqqt4/assignment_loss‚1.1643675240695852 /loss/FRt2/detection_loss‚2.2713737972947055 0loss/SRqqt3/detection_loss‚2.949728649396163 $ _timestamp‚1.788539563483362e+09 2loss/SRqqt2/assignment_loss‚1.4364239431917667 1loss/FBt1/assignment_loss‚0.37376579989924924 0loss/FBt3/assignment_loss‚0.5548174473350747 1loss/SRqqt4/detection_loss‚0.9660803257417279 .loss/FBt1/detection_loss‚2.269122736321555 /loss/FBt3/detection_loss‚1.7724017818336901 /loss/FRt1/assignment_loss‚4.424860842373906 0loss/FRt2/assignment_loss‚2.6670572549797766 0loss/FRt3/assignment_loss‚4.6977017704321415 /loss/FBt4/assignment_loss‚2.963263750076294 0loss/SRqqt2/detection_loss‚2.242071044445038 /loss/FBt2/detection_loss‚1.0778206337105345 epoch‚2 trainer/global_step‚3099 _runtime‚ 4358.79429881 _step‚253×UP[²Õ 1loss/FBt2/assignment_loss‚0.20118533609981856 1loss/SRqqt1/detection_loss‚2.4743239264935255 .loss/FBt4/detection_loss‚7.022137641906738 1loss/SRqqt3/assignment_loss‚2.546648622705386 0loss/FRt4/assignment_loss‚1.5553013256617956 1loss/SRqqt1/assignment_loss‚2.089624732732773 2loss/SRqqt4/assignment_loss‚1.1643675240695852 /loss/FRt2/detection_loss‚2.2713737972947055 0loss/SRqqt3/detection_loss‚2.949728649396163 $ _timestamp‚1.788539563483362e+09 2loss/SRqqt2/assignment_loss‚1.4364239431917667 1loss/FBt1/assignment_loss‚0.37376579989924924 0loss/FBt3/assignment_loss‚0.5548174473350747 1loss/SRqqt4/detection_loss‚0.9660803257417279 .loss/FBt1/detection_loss‚2.269122736321555 /loss/FBt3/detection_loss‚1.7724017818336901 /loss/FRt1/assignment_loss‚4.424860842373906 0loss/FRt2/assignment_loss‚2.6670572549797766 0loss/FRt3/assignment_loss‚4.6977017704321415 /loss/FBt4/assignment_loss‚2.963263750076294 0loss/SRqqt2/detection_loss‚2.242071044445038 /loss/FBt2/detection_loss‚1.0778206337105345 epoch‚2 trainer/global_step‚3099 _runtime‚ 4358.79429881 _step‚253 .loss/FRt1/detection_loss‚2.342750992701978 /loss/FRt3/detection_loss‚2.3649775106174755 /loss/FRt4/detection_loss‚1.8630534540861845 %loss/total_loss‚54.29086084410688ý–j’h“³‚(:ˆ ±åëÔŽ‘Î gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü0Cx´‚(:í ±åëÔÈ•µÐ$ disk.nvme0c0n1.in‚10467.58984375' disk.nvme0c0n1.out‚86470.9365234375 disk.dm-0.in‚10467.58984375 disk.dm-1.in‚0$ disk./.usageGB‚1179.061019897461& proc.memory.availableMB‚ 609.984375 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078" disk.nvme0n1.in‚10467.58984375 disk.sda3.out‚ 148.80859375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.625 disk.sda.out‚ 148.80859375 disk.sda1.out‚0$ memory_percent‚96.27695083618164 cpu‚25.368167900739653 network.recv‚ 1.461425e+06% disk.nvme0n1.out‚86470.9365234375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0" disk.dm-0.out‚86470.9365234375 disk.dm-1.out‚ 148.80859375 proc.cpu.threads‚17 network.sent‚ 7.492334e+06) disk./.usagePercent‚79.12777434340826 disk.sda2.in‚0 disk.sda2.out‚0¾â¦“µ‚(:ˆ ÀåëÔƒÅŽË gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚170.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W@®w¶‚(:ì ÀåëÔÝ ‡Ì( disk./.usagePercent‚79.1206140686831 disk.sda2.out‚0" disk.nvme0n1.in‚10467.58984375 disk.sda1.out‚0 disk.dm-0.in‚10467.58984375" disk.dm-0.out‚86715.4033203125 disk.dm-1.out‚ 149.390625 network.recv‚ 1.467362e+06 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚86717.2783203125% disk.nvme0n1.out‚86717.7783203125 disk.sda3.out‚ 149.390625 disk.sda.in‚0 disk.sda.out‚ 149.390625 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.62890625 disk.dm-1.in‚0$ memory_percent‚96.27695083618164 cpu‚25.367424233184934% disk./.usageGB‚1178.9543266296387$ disk.nvme0c0n1.in‚10467.58984375 disk.sda3.in‚0& proc.memory.availableMB‚ 609.984375 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 proc.cpu.threads‚17 network.sent‚ 7.600974e+06~R»ëž·‚(:“ ÏåëÔúÞžË gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.487' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.097399999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ‹G|¸‚(:ñ ÏåëÔÛ¨Í disk.sda2.out‚0$ memory_percent‚96.27697467803955 cpu‚25.366619054753208 proc.cpu.threads‚17% disk./.usageGB‚1178.9532470703125 disk.sda3.out‚ 150.1171875 disk.sda.in‚0 disk.dm-1.out‚ 150.1171875 disk.drbd1260.in‚0) proc.memory.percent‚18.75934600830078 network.recv‚ 1.472135e+06( disk./.usagePercent‚79.1205416185509" disk.nvme0n1.in‚10467.58984375% disk.nvme0n1.out‚87062.8837890625 disk.sda3.in‚0$ disk.nvme0c0n1.in‚10467.58984375' disk.nvme0c0n1.out‚87062.8837890625 proc.memory.rssMB‚ 3073.53125 disk.sda.out‚ 150.1171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚10467.58984375" disk.dm-0.out‚87062.8837890625 disk.sda2.in‚0 disk.drbd1260.out‚ 0.62890625( proc.memory.availableMB‚ 609.98046875 network.sent‚ 7.607097e+06 disk.dm-1.in‚0lsHž¹‚(:“ ÞåëÔ¢À÷Ë gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.454' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.890800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æI>Yº‚(:ô ÞåëÔ¡ÌÄÍ( disk.nvme0c0n1.out‚87428.52197265625 disk.dm-0.in‚10467.58984375 disk.dm-1.out‚ 151.1328125 cpu‚25.365775620988718) disk./.usagePercent‚79.12608238961447% disk./.usageGB‚1179.0358085632324 disk.drbd1260.in‚0" disk.nvme0n1.in‚10467.58984375 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 3073.53125) proc.memory.percent‚18.75934600830078 disk.sda3.out‚ 151.1328125 network.sent‚ 7.612277e+06 network.recv‚ 1.475608e+06 disk.sda2.out‚0$ disk.nvme0c0n1.in‚10467.58984375& disk.nvme0n1.out‚87428.52197265625 disk.sda.in‚0 disk.sda.out‚ 151.1328125# disk.dm-0.out‚87428.52197265625 disk.dm-1.in‚0$ memory_percent‚96.27697467803955( proc.memory.availableMB‚ 609.98046875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.6328125ƒ"“2»- _wandbruntime‚4419  _runtime‚4419bìLVǼÁ & lr-AdamW/pg1‚0.0015567127854753345 & lr-AdamW/pg2‚0.0015567127854753345 trainer/global_step‚3149 % _timestamp‚1.7885396241131446e+09 _runtime‚4419.424139282 _step‚254N=píÌ½Æ trainer/global_step‚3149 % _timestamp‚1.7885396241131446e+09 _runtime‚4419.424139282 _step‚254 & lr-AdamW/pg1‚0.0015567127854753345 & lr-AdamW/pg2‚0.0015567127854753345þ‰L2¾- _wandbruntime‚4419  _runtime‚4419daMxò'¿ìO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267556657472000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58246505 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291321695166976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291321671226880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255280946682608 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79532755 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65464768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚255 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258863320443711 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79559098 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144997523 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263960833427968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3595806041088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267555729498112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3594896070144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3594896070144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295080343678976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263960851430912 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79559098 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚3149 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267556657472000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79532755 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19029895 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263960833427968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19030530 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263960851430912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65464770 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7885396241146479e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267555729498112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145023868 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65464770 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3581448746743 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65464768 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267555729498112 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248528384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145023868 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65464770 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3581448746743 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65464768 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291321695166976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291321671226880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255280946682608 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79532755 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65464768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚255 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚258863320443711 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79559098 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224588288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144997523 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263960833427968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3595806041088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚267555729498112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3594896070144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3594896070144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944ÿe+Š[2Á- _wandbruntime‚4420  _runtime‚4420#ywŸü&ÂöM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79557586 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145068885 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255359981316512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19035552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79583955 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3760061926400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267639398031360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3596927514624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58264857 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65484928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚264042470516736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3596927514624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39228675 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291411896931328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚4420.628797497 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267639398031360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145042514 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295171723796992 % _timestamp‚1.7885396253165228e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295171982797824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3596017530368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65484928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3596017530368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19036180 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145042514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267638470044160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79583955 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258942530297173 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264042452513792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65484930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65484930 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3759826865664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258943455311741 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58264227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39228677 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291411920871424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255359964277152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264042452513792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚3149…Mxq'ÃûM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3596017530368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65484928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3596017530368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19036180 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145042514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267638470044160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79583955 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258942530297173 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚264042452513792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65484930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65484930 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3759826865664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258943455311741 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58264227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39228677 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291411920871424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255359964277152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264042452513792 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 trainer/global_step‚3149 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79557586 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145068885 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255359981316512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19035552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _step‚256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79583955 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3760061926400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267639398031360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3596927514624 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58264857 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65484928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚264042470516736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3596927514624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39228675 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291411896931328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚4420.628797497 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267639398031360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145042514 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295171723796992 % _timestamp‚1.7885396253165228e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295171982797824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3651893466112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66472768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147276770 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39819860 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271692771214336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3650983495168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚271693699188224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80804000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295832956449280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262869152411659 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147250425 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268041787719168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3651893466112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19321287 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268041787719168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59141149 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19320654 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262870077426019 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80777657 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295832980389376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3638233976851 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3650983495168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66472768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3817346732032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3199 _runtime‚4481.461582555 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80777657 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271692771214336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268041805722112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66472770 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885396861502006e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271693699188224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39819862 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59140514 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3817567126016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3637326001851 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80804000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268041805722112…Vuþ÷'ÔñO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268041805722112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259231843449168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3651893466112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66472768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147276770 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39819860 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚271692771214336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3650983495168 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚271693699188224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80804000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295832956449280 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚262869152411659 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147250425 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268041787719168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3651893466112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19321287 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268041787719168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59141149 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19320654 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262870077426019 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80777657 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295832980389376 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3638233976851 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3650983495168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66472768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3817346732032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3199 _runtime‚4481.461582555 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80777657 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚271692771214336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268041805722112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66472770 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885396861502006e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚271693699188224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39819862 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59140514 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3817567126016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3637326001851 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80804000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570ƒ¶¶ßŒ2Õ- _wandbruntime‚4482  _runtime‚4482­ùš…ü&ÖöM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39831925 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66492930 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3653014939648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3653014939648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚271775511760384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80828857 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19326368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299741950671360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3638443275129 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885396873624506e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299741697961984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80802488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3652104955392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271776439747584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147295416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3818515808256 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259310861043712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147321787 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268123424807936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66492930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚262949287279481 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259310844004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59158293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271776439747584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262950212294049 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚4482.676009662 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3639351250337 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147321787 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66492928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268123424807936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3652104955392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259310844004352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59158293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚271776439747584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262950212294049 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 _runtime‚4482.676009662 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3639351250337 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147321787 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66492928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268123424807936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3652104955392 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3653014939648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚271775511760384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80828857 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19326368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299741950671360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3638443275129 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885396873624506e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299741697961984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80802488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3652104955392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚271776439747584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147295416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3818515808256 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259310861043712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147321787 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268123424807936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66492930 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚262949287279481 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275829812930560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272122760013312 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40423110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60035214 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3694111231959 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40423112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272122742010368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82048902 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272122760013312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275830740904448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263182706137008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19612104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304220363163648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3707980891136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚263 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67480770 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304220617983488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3249 _runtime‚4541.652153602 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185À«É¬€èñO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300344241671680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40423110 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60035214 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3694111231959 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40423112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚272122742010368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82048902 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272122760013312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275830740904448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263182706137008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19612104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304220363163648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3707980891136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 _step‚263 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67480770 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304220617983488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3249 _runtime‚4541.652153602 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67480768 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82022559 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3707070920192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67480768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3707070920192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82048902 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300344265611776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275829812930560 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272122760013312 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149548318 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82073759 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272204379099136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877513561088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263261723731552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275912553476608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3877288986112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚264 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3709102364672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515072 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3708192380416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67500930 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60052968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149574689 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67500930 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60053581 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266956044261789 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚3249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275913481463808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40435175 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300434467376128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272204361096192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40435177 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19618404 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3695228505445 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694320530237 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0IyLÜ'ëúM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574976 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149548318 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82073759 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272204379099136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3877513561088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263261723731552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚275912553476608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3877288986112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 _step‚264 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3709102364672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248515072 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3708192380416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67500930 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60052968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149574689 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67500930 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60053581 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266956044261789 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 trainer/global_step‚3249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275913481463808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40435175 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300434467376128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272204361096192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40435177 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19618404 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3695228505445 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3694320530237 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275913481463808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19617793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 _runtime‚ 4542.99043771 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3709102364672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 % _timestamp‚1.7885397476786742e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67500928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82047390 Bøò'ûìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279966854646784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267133602903568 trainer/global_step‚3299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279967782620672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276203696301568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60929386 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151782574 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279966854646784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68488770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60928764 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83267461 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3764068316160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3749988487067 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3749080512067 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3764068316160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68488768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83293804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19902404 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151756229 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _step‚267 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3763158345216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3935125469696 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83293804 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚4602.560272476 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19903024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308790440358400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235945472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3763158345216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270883591390635 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270882666376275 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83267461 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279967782620672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267133585864208 % _timestamp‚1.7885398072491531e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276203696301568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41026360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276203714304512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68488770 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005376Gâ)n÷'üñO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151782574 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279966854646784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68488770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60928764 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83267461 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3764068316160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3749988487067 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3749080512067 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3764068316160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68488768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83293804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19902404 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151756229 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _step‚267 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3763158345216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3935125469696 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83293804 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚4602.560272476 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19903024 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308790440358400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235945472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3763158345216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270883591390635 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚270882666376275 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83267461 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279967782620672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267133585864208 % _timestamp‚1.7885398072491531e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276203696301568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41026360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276203714304512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68488770 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212005376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279966854646784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267133602903568 trainer/global_step‚3299 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279967782620672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276203696301568 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60929386 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784‹Sð©2ý- _wandbruntime‚4603  _runtime‚4603ºóŽü&þöM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276285333390336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304945776538624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 % _timestamp‚1.7885398084291508e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68508928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3936305865216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151801220 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19908151 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304945752598528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83318661 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308882082403840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270963726258665 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270962801244097 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3751105760553 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60946576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41038427 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3750197785345 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3764279805440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151827591 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151827591 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276285315387392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3765189789696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19908783 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68508930 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60947210 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3764279805440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276285333390336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3765189789696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83292292 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3299 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276285315387392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280049595192832 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68508930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83318661 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0¦døsÿûM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151827591 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151827591 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276285315387392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3765189789696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19908783 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68508930 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60947210 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3764279805440 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276285333390336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3765189789696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83292292 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3299 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228769280 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276285315387392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280049595192832 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68508930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83318661 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276285333390336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304945776538624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 % _timestamp‚1.7885398084291508e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68508928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3936305865216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151801220 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19908151 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304945752598528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83318661 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308882082403840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270963726258665 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270962801244097 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3751105760553 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60946576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41038427 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3750197785345 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3764279805440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570Œüñ‰+2€- _wandbruntime‚4603  _runtime‚4603X‡ð•SÍ 0loss/SRqqt3/detection_loss‚2.879772825241089 1loss/SRqqt4/detection_loss‚0.9577525347875532 _runtime‚4603.744707791 .loss/FBt3/detection_loss‚2.250303325292311 &loss/total_loss‚55.328588116793064 1loss/SRqqt1/assignment_loss‚1.833827720534417 2loss/SRqqt4/assignment_loss‚1.0885752242982618 /loss/FRt3/detection_loss‚2.9743452626653015 /loss/FBt1/detection_loss‚2.5264973918596905 0loss/FBt1/assignment_loss‚0.4858177810771546 .loss/FRt4/detection_loss‚1.635505503262667 0loss/SRqqt1/detection_loss‚2.565821966817302 % _timestamp‚1.7885398084337647e+09 /loss/FBt4/assignment_loss‚2.634637713432312 0loss/FRt2/assignment_loss‚3.1390968237604415 1loss/SRqqt3/assignment_loss‚2.345964468717575 0loss/SRqqt2/detection_loss‚2.267957174352237 .loss/FRt1/detection_loss‚2.390841082120553 _step‚269 /loss/FBt2/detection_loss‚1.1838348935235221 /loss/FRt2/detection_loss‚2.4382059616701945 .loss/FRt1/assignment_loss‚4.15839731387603 /loss/FRt3/assignment_loss‚4.850346837043762 0loss/FBt3/assignment_loss‚0.5044570079959445 /loss/FRt4/assignment_loss‚1.822501178173458 1loss/FBt2/assignment_loss‚0.07466399915728343 .loss/FBt4/detection_loss‚6.155410051345825 epoch‚2 trainer/global_step‚3299 2loss/SRqqt2/assignment_loss‚2.1640540757881745A›‹X‚Ò _step‚269 /loss/FBt2/detection_loss‚1.1838348935235221 /loss/FRt2/detection_loss‚2.4382059616701945 .loss/FRt1/assignment_loss‚4.15839731387603 /loss/FRt3/assignment_loss‚4.850346837043762 0loss/FBt3/assignment_loss‚0.5044570079959445 /loss/FRt4/assignment_loss‚1.822501178173458 1loss/FBt2/assignment_loss‚0.07466399915728343 .loss/FBt4/detection_loss‚6.155410051345825 epoch‚2 trainer/global_step‚3299 2loss/SRqqt2/assignment_loss‚2.1640540757881745 0loss/SRqqt3/detection_loss‚2.879772825241089 1loss/SRqqt4/detection_loss‚0.9577525347875532 _runtime‚4603.744707791 .loss/FBt3/detection_loss‚2.250303325292311 &loss/total_loss‚55.328588116793064 1loss/SRqqt1/assignment_loss‚1.833827720534417 2loss/SRqqt4/assignment_loss‚1.0885752242982618 /loss/FRt3/detection_loss‚2.9743452626653015 /loss/FBt1/detection_loss‚2.5264973918596905 0loss/FBt1/assignment_loss‚0.4858177810771546 .loss/FRt4/detection_loss‚1.635505503262667 0loss/SRqqt1/detection_loss‚2.565821966817302 % _timestamp‚1.7885398084337647e+09 /loss/FBt4/assignment_loss‚2.634637713432312 0loss/FRt2/assignment_loss‚3.1390968237604415 1loss/SRqqt3/assignment_loss‚2.345964468717575 0loss/SRqqt2/detection_loss‚2.267957174352237 .loss/FRt1/detection_loss‚2.390841082120553wF“ƒ‚(:ˆ ¡çëÔŒ¢©Ë gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚129.927' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ”{‚„‚(:÷ ¡çëÔÔº¥Î% disk./.usageGB‚1179.6260795593262 disk.sda2.out‚0" disk.nvme0n1.in‚10928.15234375 proc.memory.rssMB‚ 2978.6875 cpu‚25.355972740304786 network.sent‚ 8.004926e+06) disk./.usagePercent‚79.16569597143267 disk.dm-0.in‚10928.15234375 disk.dm-1.in‚0 disk.dm-1.out‚ 160.11328125$ disk.nvme0c0n1.in‚10928.15234375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.66015625( disk.nvme0c0n1.out‚91341.55908203125 disk.sda3.in‚0 disk.sda.out‚ 160.11328125 network.recv‚ 1.532657e+06# disk.dm-0.out‚91341.55908203125& disk.nvme0n1.out‚91341.55908203125 disk.sda3.out‚ 160.11328125 disk.sda.in‚0$ memory_percent‚95.71070671081543' proc.memory.availableMB‚ 702.7578125* proc.memory.percent‚18.180465698242188 …‚(:’ °çëÔñû„Í gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.74' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.948000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154009131 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274890348372943 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84538706 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284103896363008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280284668595712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20192751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313360608565248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3804957767175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41629612 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309366812116480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274889423358583 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284103896363008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3820155741184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309366836056576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271084465591408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820155741184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284104824336896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313360347453952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154035476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261111296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3805865742175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3349 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61822361 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69496768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3993772508672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3993535337472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84538706 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41629610 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280284650592768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20193382 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284104824336896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3819245770240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237171200 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 % _timestamp‚1.7885398693501408e+09 _step‚271 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280284650592768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154009131 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes‚}’Î.all.allocated‚274890348372943 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84538706 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284103896363008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚280284668595712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20192751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313360608565248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3804957767175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41629612 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309366812116480 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274889423358583 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚284103896363008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3820155741184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309366836056576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271084465591408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3820155741184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284104824336896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313360347453952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154035476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261111296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3805865742175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3349 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61822361 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185<¥Øä‘‚(:„ ÝçëÔà®Ì gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.9' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}†‡še’‚(:Ú ÝçëÔõ™¶Î disk.nvme0n1.in‚11059 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0 network.recv‚ 1.550562e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 161.83984375 disk.sda3.out‚ 161.83984375 proc.memory.rssMB‚ 2978.6875* proc.memory.percent‚18.180465698242188 cpu‚25.35303102040878 proc.cpu.threads‚17$ disk./.usageGB‚1180.127758026123 disk.dm-0.in‚11059 disk.sda2.in‚0( disk.nvme0c0n1.out‚92672.54248046875 disk.sda.out‚ 161.83984375 disk.sda1.in‚0$ memory_percent‚95.71375846862793' proc.memory.availableMB‚ 702.2578125 network.sent‚ 8.130174e+06 disk.drbd1260.out‚ 0.66796875 disk.nvme0c0n1.in‚11059& disk.nvme0n1.out‚92672.54248046875) disk./.usagePercent‚79.19936403427567# disk.dm-0.out‚92672.54248046875 disk.sda2.out‚0 disk.drbd1260.in‚0UÀˆ42“- _wandbruntime‚4665  _runtime‚4665S­b·ü&”öM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154080493 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3820367230464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69516930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274970483240973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61840230 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84537194 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3821277214720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84563563 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚4665.772707321 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69516928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3994701007872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284187564896256 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154054122 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154080493 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84537194 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20198555 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84563563 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309457037820928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271163483185952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 _runtime‚4665.772707321 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69516928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3994701007872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284187564896256 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154054122 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154080493 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84537194 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20198555 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84563563 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309457037820928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271163483185952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚272 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61840230 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84537194 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3821277214720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84563563 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252709376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288241866053120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3875333195264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278896180340891 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313878121278976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62716859 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85783608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70504770 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284365622886912 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3860835022283 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288240938079232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3876243166208 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42232860 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4052412464128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚4727.852105948 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156288378 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 $ _timestamp‚1.788539932541083e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284365604883968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313878097338880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62716223 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3876243166208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570#wôö'¦ðO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85783608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚288240938079232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317930533743104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284365604883968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62716859 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85783608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70504770 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284365622886912 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3860835022283 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288240938079232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3876243166208 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42232860 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4052412464128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _runtime‚4727.852105948 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156288378 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 $ _timestamp‚1.788539932541083e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284365604883968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313878097338880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62716223 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3876243166208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317930278923264 trainer/global_step‚3399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70504768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230879744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275035362357968 _step‚275 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3861742997283 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284365622886912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20483997 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4052181584384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275035345318608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42232862 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3875333195264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156262033 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288241866053120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3875333195264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278896180340891 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313878121278976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42244925 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275114379952512 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85782096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156307024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3862860270769 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85782096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284447241972736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3877364639744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876454655488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284447223969792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚284447241972736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318021925824512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313968346983424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70524930 _step‚276 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288323678625280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62734731 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62734090 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70524930 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156333395 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313968323043328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3399 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4053356363264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284447223969792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20489165 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3861952295561l%]©ûM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2048980ÙÙǤ&4 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3876454655488 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275114362913152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284447223969792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20489165 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3861952295561 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85808465 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70524928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278976315208713 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318021679406592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4728.900444588 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42244925 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275114379952512 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85782096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156307024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3862860270769 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85782096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284447241972736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3877364639744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876454655488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚284447223969792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚284447241972736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚318021925824512 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313968346983424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70524930 _step‚276 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288323678625280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62734731 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62734090 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70524930 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156333395 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚313968323043328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3399 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4053356363264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FL&x´‚(:í ÕèëÔÇ´ÅÐ network.recv‚ 1.588882e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 165.390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.68359375) disk./.usagePercent‚79.17306412427564" disk.dm-0.out‚95005.8798828125$ disk.nvme0c0n1.in‚11341.90234375' disk.nvme0c0n1.out‚95005.8798828125# memory_percent‚95.7129955291748* proc.memory.percent‚18.180465698242188 cpu‚25.347243228097014 proc.cpu.threads‚17 network.sent‚ 8.448066e+06% disk./.usageGB‚1179.7358703613281 disk.sda3.in‚0 disk.sda.out‚ 165.390625 disk.sda1.in‚0 disk.sda3.out‚ 165.390625 disk.sda1.out‚0 disk.dm-0.in‚11341.90234375" disk.nvme0n1.in‚11341.90234375% disk.nvme0n1.out‚95005.8798828125' proc.memory.availableMB‚ 702.3828125 proc.memory.rssMB‚ 2978.6875ßÅòØ2µ- _wandbruntime‚4788  _runtime‚4788>qƶÀ & lr-AdamW/pg1‚0.0015153026392676898 & lr-AdamW/pg2‚0.0015153026392676898 trainer/global_step‚3449 % _timestamp‚1.7885399933314984e+09 _runtime‚ 4788.64259247 _step‚278T÷iCË·Å _step‚278 & lr-AdamW/pg1‚0.0015153026392676898 & lr-AdamW/pg2‚0.0015153026392676898 trainer/global_step‚3449 % _timestamp‚1.7885399933314984e+09 _runtime‚ 4788.64259247–]1¾¦2¸- _wandbruntime‚4788  _runtime‚4788´p³ò'¹ìO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87002167 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278986242085168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292378907769344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3932330591232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3917620252391 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288446577178112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3932330591232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4111305366016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4111087069184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288446559175168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282902937323199 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282903862337559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42836110 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87028510 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20776016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322500711867392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63612126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚4788.644574637 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292377979795456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158541280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20776647 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3931420620288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318389382561280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚3449 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71512770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3931420620288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42836112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3916712277391 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278986225045808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87028510 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885399933330204e+09…>yóÁºñO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242236928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278986225045808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚263è_qü645 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87028510 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 % _timestamp‚1.7885399933330204e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87002167 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278986242085168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292378907769344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3932330591232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3917620252391 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚288446577178112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3932330591232 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4111305366016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4111087069184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚288446559175168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282902937323199 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282903862337559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42836110 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87028510 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20776016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 _step‚279 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322500711867392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63612126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218296832 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 _runtime‚4788.644574637 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292377979795456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158541280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20776647 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3931420620288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318389382561280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚3449 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71512770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3931420620288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42836112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3916712277391 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0—vªæù2»- _wandbruntime‚4790  _runtime‚4790ølÐü&¼öM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63629975 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚3449 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚280 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288528178260992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3933452064768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158586297 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318479632205824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318479608265728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288528196263936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42848177 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288528178260992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282983997205589 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292461648328704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3932542080512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4112490568192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71532928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87026998 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87053367 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44'"½ '½ûM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158586297 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216186368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158559926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158586297 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318479632205824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318479608265728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288528196263936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42848177 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚288528178260992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282983997205589 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292461648328704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3932542080512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4112490568192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71532928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87026998 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87053367 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63629975 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 trainer/global_step‚3449 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚280 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288528178260992 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3933452064768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3988418016256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88247069 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4169977792512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 $ _timestamp‚1.788540053010494e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64508034 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3987508045312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72520770 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72520768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43439360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296515021511680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21068674 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64508698 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21069336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3988418016256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43439362 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292527513466368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322900667783680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296515949485568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286910619319867 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241365504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4170219158016 _runtime‚4848.321514519 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72520770 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292527531469312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296515949485568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88273412 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160767837 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282937121812368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚282937104773008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292527513466368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286909694305507 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296515021511680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21068674 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64508698 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21069336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3988418016256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43439362 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292527513466368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322900667783680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296515949485568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286910619319867 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241365504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4170219158016 _runtime‚4848.321514519 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72520770 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292527531469312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296515949485568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88273412 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160767837 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚282937121812368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚282937104773008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚292527513466368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286909694305507 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3988418016256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88247069 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4169977792512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 $ _timestamp‚1.788540053010494e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64508034 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3987508045312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72520770 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72520768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43439360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283016139406912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88298269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43451425 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21075154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 trainer/global_step‚3499 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64525943 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296598690044928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160839199 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322990893488128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4171171199488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72540928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322990917428224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3988629505536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21074518 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327162309008384 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88298269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3973706805777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚284 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160839199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296598690044928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3989539489792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚292609132552192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286990754187897 _runtime‚4849.287026031 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43451427 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚6369eßðÑûM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327162064687616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1£K™A& B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283016139406912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88298269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43451425 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21075154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 trainer/global_step‚3499 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64525943 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296598690044928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160839199 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322990893488128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4171171199488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72540928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322990917428224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244320768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3988629505536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21074518 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327162309008384 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88298269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380672 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3973706805777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚284 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160839199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296598690044928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3989539489792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚292609132552192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286990754187897 _runtime‚4849.287026031 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43451427 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64526581œÄ×ây2Ò- _wandbruntime‚4849  _runtime‚4849ÒÑzRÓÌ 0loss/FRt4/assignment_loss‚2.1214087036522953 1loss/SRqqt3/assignment_loss‚2.122108517490603 /loss/FBt4/assignment_loss‚2.752443552017212 0loss/SRqqt1/detection_loss‚2.667009457089007 1loss/SRqqt4/detection_loss‚0.9818741990327835 /loss/FBt2/detection_loss‚0.9336615834917341 /loss/FRt2/assignment_loss‚2.751977725940592 /loss/FBt3/assignment_loss‚0.785163647354193 /loss/FBt3/detection_loss‚2.2203580934554337 .loss/FBt4/detection_loss‚6.631519794464111 _runtime‚4849.288474883 _step‚285 /loss/FRt3/detection_loss‚2.4790985697233805 1loss/SRqqt3/detection_loss‚2.4939820369084678 epoch‚2 1loss/SRqqt1/assignment_loss‚2.325780816078186 2loss/SRqqt2/assignment_loss‚1.6383094292134046 0loss/FBt2/assignment_loss‚0.2240029016004295 .loss/FBt1/detection_loss‚2.239758551120758 $ _timestamp‚1.788540053977714e+09 0loss/FRt1/assignment_loss‚3.6235156848150143 2loss/SRqqt4/assignment_loss‚1.1529302856493742 .loss/FRt1/detection_loss‚2.704151832005557 /loss/FRt4/detection_loss‚2.3264750057662074 0loss/FBt1/assignment_loss‚0.4557303978037089 /loss/FRt3/assignment_loss‚4.061399225781604 1loss/SRqqt2/detection_loss‚2.6923766255378725 $loss/total_loss‚54.7663824057111 trainer/global_step‚3499 .loss/FRt2/detection_loss‚2.381345769719166¯ñfbWÔÑ 1loss/SRqqt4/detection_loss‚0.9818741990327835 /loss/FBt2/detection_loss‚0.9336615834917341 /loss/FRt2/assignment_loss‚2.751977725940592 /loss/FBt3/assignment_loss‚0.785163647354193 /loss/FBt3/detection_loss‚2.2203580934554337 .loss/FBt4/detection_loss‚6.631519794464111 _runtime‚4849.288474883 _step‚285 /loss/FRt3/detection_loss‚2.4790985697233805 1loss/SRqqt3/detection_loss‚2.4939820369084678 epoch‚2 1loss/SRqqt1/assignment_loss‚2.325780816078186 2loss/SRqqt2/assignment_loss‚1.6383094292134046 0loss/FBt2/assignment_loss‚0.2240029016004295 .loss/FBt1/detection_loss‚2.239758551120758 $ _timestamp‚1.788540053977714e+09 0loss/FRt1/assignment_loss‚3.6235156848150143 2loss/SRqqt4/assignment_loss‚1.1529302856493742 .loss/FRt1/detection_loss‚2.704151832005557 /loss/FRt4/detection_loss‚2.3264750057662074 0loss/FBt1/assignment_loss‚0.4557303978037089 /loss/FRt3/assignment_loss‚4.061399225781604 1loss/SRqqt2/detection_loss‚2.6923766255378725 $loss/total_loss‚54.7663824057111 trainer/global_step‚3499 .loss/FRt2/detection_loss‚2.381345769719166 0loss/FRt4/assignment_loss‚2.1214087036522953 1loss/SRqqt3/assignment_loss‚2.122108517490603 /loss/FBt4/assignment_loss‚2.752443552017212 0loss/SRqqt1/detection_loss‚2.667009457089007€=ï.“Õ‚(:ˆ  éëÔíظÑ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚149.611' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âQúvÖ‚(:ë  éëÔáÔÒ disk.sda2.out‚0( disk.nvme0c0n1.out‚96595.17626953125 disk.sda.in‚0' proc.memory.availableMB‚ 702.0234375 proc.memory.rssMB‚ 2978.6875* proc.memory.percent‚18.180465698242188 cpu‚25.343749780793807 network.sent‚ 8.579305e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.69140625! disk.nvme0c0n1.in‚ 11485.03125 disk.sda3.out‚ 167.5546875 disk.sda.out‚ 167.5546875 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚79.17306847640373 disk.nvme0n1.in‚ 11485.03125& disk.nvme0n1.out‚96595.17626953125 disk.sda3.in‚0# disk.dm-0.out‚96595.17626953125 disk.dm-1.in‚0 disk.dm-1.out‚ 167.5546875$ memory_percent‚95.71518898010254 network.recv‚ 1.611575e+06% disk./.usageGB‚1179.7359352111816 disk.sda2.in‚0 disk.dm-0.in‚ 11485.03125 proc.cpu.threads‚17Áö ’ׂ(:‡ ¯éëÔŸ‘ªÌ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.899 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ÂÃ{Ø‚(:ð ¯éëÔ¤·õÎ disk.dm-1.in‚0 disk.dm-1.out‚ 167.8671875 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6953125# disk.nvme0c0n1.in‚ 11654.6796875 disk.sda2.in‚0( disk.nvme0c0n1.out‚96892.87451171875 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0# disk.dm-0.out‚96892.87451171875! disk.nvme0n1.in‚ 11654.6796875 network.recv‚ 1.61811e+06 disk.sda.out‚ 167.8671875 disk.dm-0.in‚ 11654.6796875$ memory_percent‚95.71521282196045 proc.memory.rssMB‚ 2978.6875 cpu‚25.343046893299658 network.sent‚ 8.688221e+06% disk./.usageGB‚1179.7359924316406 disk.sda3.out‚ 167.8671875 disk.sda1.in‚0( proc.memory.availableMB‚ 702.01953125* proc.memory.percent‚18.180465698242188 proc.cpu.threads‚17& disk.nvme0n1.out‚96892.87451171875) disk./.usagePercent‚79.17307231651674Û<¯%“Ù‚(:ˆ ¾éëÔãžÄÍ gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.529' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.3058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A\Þ{Ú‚(:ð ¾éëÔÍ£ìÎ# disk.dm-0.out‚97214.46826171875 disk.dm-1.in‚0 disk.dm-1.out‚ 168.2578125# disk.nvme0c0n1.in‚ 11654.6796875( disk.nvme0c0n1.out‚97214.46826171875 disk.sda.in‚0$ memory_percent‚95.71373462677002* proc.memory.percent‚18.180465698242188 network.sent‚ 8.695157e+06) disk./.usagePercent‚79.17307794868249% disk./.usageGB‚1179.7360763549805 disk.sda2.out‚0& disk.nvme0n1.out‚97214.46826171875 disk.sda3.out‚ 168.2578125 disk.sda1.out‚0 proc.memory.rssMB‚ 2978.6875! disk.nvme0n1.in‚ 11654.6796875 disk.sda3.in‚0 disk.sda.out‚ 168.2578125 disk.sda1.in‚0( proc.memory.availableMB‚ 702.26171875 cpu‚25.34235872017178 proc.cpu.threads‚17 network.recv‚ 1.621673e+06 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6953125 disk.dm-0.in‚ 11654.6796875>yù¦žÛ‚(:“ ÍéëÔ⸖Ì gpu.0.gpu‚36.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.119799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š[•ÞmÜ‚(:â ÍéëÔÛš‰Ï" disk.dm-0.out‚97570.0205078125 disk.dm-1.out‚168.625 disk.drbd1260.in‚0$ memory_percent‚95.71373462677002 cpu‚25.34172348761503 proc.cpu.threads‚17 disk.sda1.out‚0( proc.memory.availableMB‚ 702.26171875 proc.memory.rssMB‚ 2978.6875* proc.memory.percent‚18.180465698242188) disk./.usagePercent‚79.17307666864481 disk.dm-0.in‚ 11654.6796875 disk.dm-1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 11654.6796875' disk.nvme0c0n1.out‚97570.0205078125! disk.nvme0n1.in‚ 11654.6796875% disk.nvme0n1.out‚97570.0205078125 network.sent‚ 8.697468e+06 network.recv‚ 1.624531e+06% disk./.usageGB‚1179.7360572814941 disk.sda3.in‚0 disk.sda.out‚168.625 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.69921875 disk.sda3.out‚168.625 disk.sda.in‚0i´¢2Ý- _wandbruntime‚4909  _runtime‚4909AyV€ÇÞÁ & lr-AdamW/pg2‚0.0015003622151908075 trainer/global_step‚3549 % _timestamp‚1.7885401145979855e+09 _runtime‚4909.908973411 _step‚286 & lr-AdamW/pg1‚0.0015003622151908075x ,dÌßÆ % _timestamp‚1.7885401145979855e+09 _runtime‚4909.908973411 _step‚286 & lr-AdamW/pg1‚0.0015003622151908075 & lr-AdamW/pg2‚0.0015003622151908075 trainer/global_step‚3549ž-"í®2à- _wandbruntime‚4909  _runtime‚4909–-Ið'áêO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290917376302175 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4044505441280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21361252 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4029374762607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296608467757568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327411976946176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300652063227904 _runtime‚4909.910167306 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89491971 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331641096530432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4044505441280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331640852196352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4229119584256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89518314 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89491971 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290916451287815 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300652991201792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300652991201792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73528770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21361903 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286887984500208 trainer/global_step‚3549 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4028466787607 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73528770 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73528768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44042612 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296608485760512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73528768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163020739g™ów€âïO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4028466787607 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73528770 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73528768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44042612 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296608485760512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73528768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163020739 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290917376302175 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 [IDeviceStatsc£ŽþuMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4044505441280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21361252 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4029374762607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296608467757568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244334080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327411976946176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300652063227904 _runtime‚4909.910167306 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89491971 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331641096530432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4044505441280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331640852196352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4229119584256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89518314 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89491971 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290916451287815 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300652991201792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300652991201792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73528770 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21361903 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286887984500208 trainer/global_step‚3549 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163092101 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296690086843392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327502202650624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _step‚288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296690104846336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44054675 trainer/global_step‚3549 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73548930 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21367667 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4230051219968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4044716930560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4044716930560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300734803773952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4045626914816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73548930 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73548928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327502178710528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290996586155637 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4030492036093 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4029584060885 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65421689 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163092101 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4045626914816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89516802 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65422344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89543171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296690086843392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚4910.9954380361N u'åúM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73548928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44054677 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚655 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331732493125632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788540115681981e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300735731761152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290997511170205 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239255040 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163065730 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163092101 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296690086843392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327502202650624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 _step‚288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296690104846336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44054675 trainer/global_step‚3549 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73548930 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21367667 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4230051219968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4044716930560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4044716930560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300734803773952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4045626914816 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73548930 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73548928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327502178710528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290996586155637 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4030492036093 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4029584060885 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65421689 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163092101 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4045626914816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89516802 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65422344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89543171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296690086843392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _runtime‚4910.995438036 J+~<2æ- _wandbruntime‚4910  _runtime‚4910´ÞçYçÓ 2loss/SRqqt3/assignment_loss‚1.9638945635121603 epoch‚2 % _timestamp‚1.7885401156865447e+09 /loss/FRt1/detection_loss‚2.5094502732862494 .loss/FRt2/detection_loss‚2.236099744454408 0loss/FRt2/assignment_loss‚2.9358766552729483 0loss/FRt4/assignment_loss‚1.8378077245810454 1loss/SRqqt4/assignment_loss‚1.084040450369657 0loss/FBt3/assignment_loss‚0.5790427069715633 /loss/FBt2/detection_loss‚1.0137413835122449 _step‚289 1loss/FBt2/assignment_loss‚0.13171762303227855 0loss/SRqqt2/detection_loss‚2.007940363425475 /loss/FRt3/detection_loss‚2.6612303372385213 1loss/SRqqt1/detection_loss‚2.7384084289272628 0loss/SRqqt3/detection_loss‚2.495330253472695 1loss/SRqqt4/detection_loss‚0.8969710287779118 .loss/FRt3/assignment_loss‚3.72478301525116 .loss/FBt4/detection_loss‚6.707342624664307 0loss/FRt1/assignment_loss‚3.4715155685270154 /loss/FRt4/detection_loss‚1.7239265366840888 /loss/FBt3/detection_loss‚3.2659051979426295 trainer/global_step‚3549 _runtime‚4910.997435211 /loss/FBt4/assignment_loss‚3.244004726409912 2loss/SRqqt1/assignment_loss‚1.4357692785561085 2loss/SRqqt2/assignment_loss‚1.3405987786558957 1loss/FBt1/assignment_loss‚0.35784782938373444 /loss/FBt1/detection_loss‚2.0809589976614173 %loss/total_loss‚52.44420409057069]…sI^èØ /loss/FBt3/detection_loss‚3.2659051979426295 trainer/global_step‚3549 _runtime‚4910.997435211 /loss/FBt4/assignment_loss‚3.244004726409912 2loss/SRqqt1/assignment_loss‚1.4357692785561085 2loss/SRqqt2/assignment_loss‚1.3405987786558957 1loss/FBt1/assignment_loss‚0.35784782938373444 /loss/FBt1/detection_loss‚2.0809589976614173 %loss/total_loss‚52.44420409057069 2loss/SRqqt3/assignment_loss‚1.9638945635121603 epoch‚2 % _timestamp‚1.7885401156865447e+09 /loss/FRt1/detection_loss‚2.5094502732862494 .loss/FRt2/detection_loss‚2.236099744454408 0loss/FRt2/assignment_loss‚2.9358766552729483 0loss/FRt4/assignment_loss‚1.8378077245810454 1loss/SRqqt4/assignment_loss‚1.084040450369657 0loss/FBt3/assignment_loss‚0.5790427069715633 /loss/FBt2/detection_loss‚1.0137413835122449 _step‚289 1loss/FBt2/assignment_loss‚0.13171762303227855 0loss/SRqqt2/detection_loss‚2.007940363425475 /loss/FRt3/detection_loss‚2.6612303372385213 1loss/SRqqt1/detection_loss‚2.7384084289272628 0loss/SRqqt3/detection_loss‚2.495330253472695 1loss/SRqqt4/detection_loss‚0.8969710287779118 .loss/FRt3/assignment_loss‚3.72478301525116 .loss/FBt4/detection_loss‚6.707342624664307 0loss/FRt1/assignment_loss‚3.4715155685270154 /loss/FRt4/detection_loss‚1.7239265366840888¡Õ÷â`é‚(:’ ÜéëÔ¹æÌ gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.397000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^6ìÍ|ê‚(:ñ ÜéëÔÃ÷–Ï cpu‚25.34109305503924 network.sent‚ 8.716465e+06 network.recv‚ 1.629295e+06 disk.drbd1260.out‚ 0.69921875# disk.nvme0c0n1.in‚ 11654.6796875' disk.nvme0c0n1.out‚97660.9775390625% disk.nvme0n1.out‚97660.9775390625 disk.sda.out‚ 169.05859375 disk.sda1.in‚0) disk./.usagePercent‚79.17307538860715% disk./.usageGB‚1179.7360382080078 disk.dm-1.out‚ 169.05859375 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 11654.6796875" disk.dm-0.out‚97660.9775390625 disk.sda2.in‚0 disk.sda3.out‚ 169.05859375$ memory_percent‚95.71692943572998( proc.memory.availableMB‚ 701.73828125* proc.memory.percent‚18.180465698242188 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 11654.6796875 disk.sda.in‚0 proc.memory.rssMB‚ 2978.6875<¼úÝ“ë‚(:ˆ ëéëÔ¨¹ˆÌ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚121.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[;c wì‚(:ì ëéëÔÇÿÉÐ# disk.dm-0.out‚97992.78759765625 disk.dm-1.in‚0 disk.dm-1.out‚ 169.46875 network.sent‚ 8.814224e+06$ disk./.usageGB‚1179.712532043457 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.703125$ disk.nvme0c0n1.in‚11698.33984375( proc.memory.availableMB‚ 702.00390625* proc.memory.percent‚18.180465698242188 disk.sda2.out‚0( disk.nvme0c0n1.out‚97992.78759765625 disk.sda3.in‚0 disk.sda3.out‚ 169.46875 proc.memory.rssMB‚ 2978.6875 cpu‚25.340352959303956 network.recv‚ 1.635646e+06) disk./.usagePercent‚79.17149787018116 disk.sda.out‚ 169.46875 disk.sda1.in‚0 disk.dm-0.in‚11698.33984375$ memory_percent‚95.71530818939209 proc.cpu.threads‚17" disk.nvme0n1.in‚11698.33984375& disk.nvme0n1.out‚97992.78759765625 disk.sda.in‚0 disk.sda1.out‚0ˆUQMží‚(:“ úéëÔ録Ë gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.471' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.294200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬QÀ·Yî‚(:ì úéëÔÊíÌ disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 11781.71875# disk.dm-0.out‚98314.35791015625 disk.sda.out‚ 170.24609375 proc.memory.rssMB‚ 2978.6875* proc.memory.percent‚18.180465698242188 proc.cpu.threads‚17( disk.nvme0c0n1.out‚98314.35791015625 disk.dm-1.out‚ 170.24609375 disk.drbd1260.in‚0! disk.nvme0c0n1.in‚ 11781.71875 disk.nvme0n1.in‚ 11781.71875 disk.sda3.out‚ 170.24609375 disk.sda.in‚0' proc.memory.availableMB‚ 701.7578125 network.sent‚ 8.817386e+06 network.recv‚ 1.638971e+06 disk.drbd1260.out‚0.703125& disk.nvme0n1.out‚98314.35791015625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.71681022644043 cpu‚25.339820578910327) disk./.usagePercent‚79.17149915021882% disk./.usageGB‚1179.7125511169434 disk.dóT’Qm-1.in‚0 disk.sda2.in‚0½¯æ“ï‚(:ˆ ‰êëÔ«ÅœÍ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾¨xð‚(:í ‰êëÔ÷“ÁÒ$ disk./.usageGB‚1179.712776184082 disk.sda.out‚ 171.05859375 disk.sda1.in‚0# disk.dm-0.out‚98828.56494140625 disk.sda2.in‚0 proc.memory.rssMB‚ 2978.6875* proc.memory.percent‚18.180465698242188 cpu‚25.339193480184875) disk./.usagePercent‚79.17151425466335 disk.sda3.in‚0 disk.sda3.out‚ 171.05859375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 11781.71875 disk.dm-1.in‚0 disk.drbd1260.in‚0 network.sent‚ 8.82248e+06 network.recv‚ 1.642418e+06 disk.drbd1260.out‚ 0.70703125! disk.nvme0c0n1.in‚ 11781.71875( disk.nvme0c0n1.out‚98828.56494140625 disk.nvme0n1.in‚ 11781.71875& disk.nvme0n1.out‚98828.56494140625$ memory_percent‚95.71549892425537 disk.dm-1.out‚ 171.05859375 disk.sda2.out‚0( proc.memory.availableMB‚ 701.97265625 proc.cpu.threads‚17\Ÿ¬ 2ñ- _wandbruntime‚4971  _runtime‚4971yâ‰QÇòÁ & lr-AdamW/pg1‚0.0014926856482982477 & lr-AdamW/pg2‚0.0014926856482982477 trainer/global_step‚3599 % _timestamp‚1.7885401758508737e+09 _runtime‚4971.162138342 _step‚290x¿õ'ÌóÆ _runtime‚4971.162138342 _step‚290 & lr-AdamW/pg1‚0.0014926856482982477 & lr-AdamW/pg2‚0.0014926856482982477 trainer/global_step‚3599 % _timestamp‚1.7885401758508737e+09¢ÀÓ2ô- _wandbruntime‚4971  _runtime‚4971¶m!õò'õìO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300689440051712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336211291022336 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74536770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90763216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165273641 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165299986 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90736873 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66298840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331923262168576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4100592866304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4971.164431191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336211038299648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290838864227408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90763216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74536768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44645862 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885401758529167e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300689422048768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4287800071168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚3599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304790032918016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294924133284483 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304789104944128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290838881266768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300689440051712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4100592866304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90736873 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331923238228480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4288028853760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4085252017715 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304790032918016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21652978 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294923208270123 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44645860 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74536770 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21652331 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4099682895360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4084344042715 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66298191 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74536768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300689422048768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304789104944128¨ÂÅ÷'öñO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927973888 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290838881266768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300689440051712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4100592866304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909970944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90736873 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331923238228480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4288028853760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4085252017715 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304790032918016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135737856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21652978 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294923208270123 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416517 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44645860 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74536770 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21652331 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4099682895360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4084344042715 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66298191 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74536768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚300689422048768 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304789104944128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728389 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300689440051712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336211291022336 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909970944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189767680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74536770 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90763216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165273641 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165299986 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90736873 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927973888 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66298840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331923262168576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4100592866304 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189767680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4971.164431191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚336211038299648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290838864227408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90763216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74536768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135737856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44645862 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885401758529167e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300689422048768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4287800071168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚3599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304790032918016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294924133284483 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304789104944128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚649 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0£Äm>¯2÷- _wandbruntime‚4972  _runtime‚49726V7øû&øõM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290917898861312 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4289203938816 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4100804355584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336302445393920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚3599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74556928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90788073 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4101714339840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165318632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74556930 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300771059137536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300771059137536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21658073 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295004268152513 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304872773477376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304872773477376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74556930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 _runtime‚4972.490046229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304871845490176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66315998 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44657925 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332013463932928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚332013487873024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4101714339840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66316637 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165345003 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295003343137945 $ _timestamp‚1.788540177178421e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90761704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44657927 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165345003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21658710 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290917881821952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0+_ÝÔvùúM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165345003 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295003343137945 $ _timestamp‚1.788540177178421e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90761704 VIDeviceStat¯y`÷Š%sMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44657927 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165345003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21658710 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290917881821952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290917898861312 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4289203938816 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4100804355584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336302445393920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728389 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417920 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚3599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74556928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90788073 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4101714339840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189767680 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165318632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74556930 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚300771059137536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300771059137536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21658073 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295004268152513 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304872773477376 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135737856 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189767680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304872773477376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135737856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74556930 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416517 _runtime‚4972.490046229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304871845490176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66315998 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44657925 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332013463932928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚332013487873024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4101714339840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66316637 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570¤F™rÑ2ú- _wandbruntime‚4972  _runtime‚4972nšÌPûÊ 2loss/SRqqt4/assignment_loss‚0.9824041658043862 1loss/FBt2/assignment_loss‚0.19246028896087675 .loss/FRt2/detection_loss‚2.516651018044433 1loss/SRqqt1/detection_loss‚2.6380661748922787 1loss/SRqqt3/detection_loss‚2.5823761453995338 .loss/FBt2/detection_loss‚1.326801150075851 epoch‚2 /loss/FRt1/assignment_loss‚4.111450068328692 /loss/FBt1/assignment_loss‚0.447017614701718 0loss/SRqqt2/detection_loss‚2.030728272853359 % _timestamp‚1.7885401771809952e+09 /loss/FRt3/assignment_loss‚3.460403711409182 0loss/FBt3/assignment_loss‚0.5186236170751727 .loss/FBt4/detection_loss‚6.574915885925293 2loss/SRqqt1/assignment_loss‚2.2521173793535967 1loss/SRqqt2/assignment_loss‚2.038166321543676 .loss/FRt3/detection_loss‚2.651490517564427 1loss/SRqqt4/detection_loss‚0.9109315241575241 .loss/FBt1/detection_loss‚2.611275457418882 _runtime‚4972.491783191 _step‚293 0loss/FRt4/assignment_loss‚1.9558304502413824 2loss/SRqqt3/assignment_loss‚1.8815422377620752 trainer/global_step‚3599 /loss/FRt2/assignment_loss‚2.897892017622252 /loss/FBt4/assignment_loss‚2.807835578918457 .loss/FRt4/detection_loss‚2.045181516128091 &loss/total_loss‚53.723848200326216 ,loss/FBt3/detection_loss‚1.9418617973725 .loss/FRt1/detection_loss‚2.347825288772583Í9” UüÏ ,loss/FBt3/detection_loss‚1.9418617973725 .loss/FRt1/detection_loss‚2.347825288772583 2loss/SRqqt4/assignment_loss‚0.9824041658043862 1loss/FBt2/assignment_loss‚0.19246028896087675 .loss/FRt2/detection_loss‚2.516651018044433 1loss/SRqqt1/detection_loss‚2.6380661748922787 1loss/SRqqt3/detection_loss‚2.5823761453995338 .loss/FBt2/detection_loss‚1.326801150075851 epoch‚2 /loss/FRt1/assignment_loss‚4.111450068328692 /loss/FBt1/assignment_loss‚0.447017614701718 0loss/SRqqt2/detection_loss‚2.030728272853359 % _timestamp‚1.7885401771809952e+09 /loss/FRt3/assignment_loss‚3.460403711409182 0loss/FBt3/assignment_loss‚0.5186236170751727 .loss/FBt4/detection_loss‚6.574915885925293 2loss/SRqqt1/assignment_loss‚2.2521173793535967 1loss/SRqqt2/assignment_loss‚2.038166321543676 .loss/FRt3/detection_loss‚2.651490517564427 1loss/SRqqt4/detection_loss‚0.9109315241575241 .loss/FBt1/detection_loss‚2.611275457418882 _runtime‚4972.491783191 _step‚293 0loss/FRt4/assignment_loss‚1.9558304502413824 2loss/SRqqt3/assignment_loss‚1.8815422377620752 trainer/global_step‚3599 /loss/FRt2/assignment_loss‚2.897892017622252 /loss/FBt4/assignment_loss‚2.807835578918457 .loss/FRt4/detection_loss‚2.045181516128091 &loss/total_loss‚53.723848200326216¥¿³“ý‚(:ˆ ˜êëÔðî›Ñ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.699' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ª½Övþ‚(:ë ˜êëÔó¤þÝ( disk.nvme0c0n1.out‚99231.85400390625 disk.sda3.out‚ 172.1640625$ memory_percent‚95.71568965911865( proc.memory.availableMB‚ 701.94140625 cpu‚25.338510230201514% disk./.usageGB‚1179.7362632751465 disk.sda.out‚ 172.1640625 disk.sda1.out‚0 disk.nvme0n1.in‚ 11781.71875& disk.nvme0n1.out‚99231.85400390625 proc.memory.rssMB‚ 2978.6875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-0.in‚ 11781.71875 disk.dm-1.in‚0 disk.sda3.in‚0* proc.memory.percent‚18.180465698242188 network.recv‚ 1.64692e+06) disk./.usagePercent‚79.17309049305167 disk.sda.in‚0 disk.dm-1.out‚ 172.1640625 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.70703125! disk.nvme0c0n1.in‚ 11781.71875 network.sent‚ 8.828443e+06# disk.dm-0.out‚99231.85400390625 disk.sda2.out‚0Ä\.“ÿ‚(:ˆ §êëÔ£ð„Ì gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.633' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rGav€‚(:ë §êëÔœœçÌ* proc.memory.percent‚18.180465698242188 cpu‚25.337929217207254 disk.sda2.in‚0% disk./.usageGB‚1179.7363014221191 disk.sda1.out‚0 disk.dm-0.in‚ 11781.71875 disk.dm-1.in‚0 disk.dm-1.out‚ 172.99609375 disk.nvme0n1.in‚ 11781.71875 disk.sda3.out‚ 172.99609375 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.7109375 proc.memory.rssMB‚ 2978.6875 proc.cpu.threads‚17 network.sent‚ 8.939316e+06 disk.sda.out‚ 172.99609375 disk.sda1.in‚0 disk.drbd1260.in‚0! disk.nvme0c0n1.in‚ 11781.71875& disk.nvme0n1.out‚99409.47705078125 disk.sda3.in‚0$ memory_percent‚95.71423530578613 network.recv‚ 1.653626e+06' disk./.usagePercent‚79.173093053127# disk.dm-0.out‚99409.47705078125( disk.nvme0c0n1.out‚99409.47705078125' proc.memory.availableMB‚ 702.1796875Ç‹Ðt’‚(:‡ ¶êëÔòÌ­Ê gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7546 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£9ò©|‚‚(:ñ ¶êëÔáÁ±Í disk.sda2.in‚0 disk.sda3.out‚ 173.80078125 disk.sda1.out‚0# disk.dm-0.out‚99472.21533203125% proc.memory.availableMB‚ 622.71875* proc.memory.percent‚18.663406372070312 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7109375# disk.nvme0c0n1.in‚ 11933.2109375! disk.nvme0n1.in‚ 11933.2109375 disk.sda1.in‚0$ memory_percent‚96.19922637939453 network.sent‚ 8.942471e+06 network.recv‚ 1.657206e+06) disk./.usagePercent‚79.17309023704414% disk./.usageGB‚1179.7362594604492& disk.nvme0n1.out‚99472.21533203125 disk.sda.in‚0 disk.dm-0.in‚ 11933.2109375 disk.dm-1.out‚ 173.80078125( disk.nvme0c0n1.out‚99472.21533203125 disk.sda3.in‚0 disk.sda.out‚ 173.80078125 disk.dm-1.in‚0 proc.memory.rssMB‚ 3057.8125 cpu‚25.343148242638513„dÙ;žƒ‚(:“ ÅêëÔ¿áÅÏ gpu.0.gpu‚58.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ ßæ{„‚(:ð ÅêëÔÇ·‹Ò disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 11933.2109375& disk.nvme0n1.out‚99832.67626953125 disk.sda3.out‚ 174.26953125 disk.sda1.in‚0 network.sent‚ 8.94763e+06 network.recv‚ 1.660664e+06) disk./.usagePercent‚79.17151399865581# disk.dm-0.out‚99832.67626953125 disk.dm-1.in‚0% proc.memory.availableMB‚ 622.96875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚99832.67626953125 disk.sda3.in‚0 disk.sda.out‚ 174.26953125 disk.sda1.out‚0 disk.dm-0.in‚ 11933.2109375 disk.dm-1.out‚ 174.26953125 proc.memory.rssMB‚ 3057.8125* proc.memory.percent‚18.663406372070312 disk.drbd1260.out‚ 0.71484375# disk.nvme0c0n1.in‚ 11933.2109375 disk.sda.in‚0$ memory_percent‚96.19770050048828 cpu‚25.35543865755134% disk./.usageGB‚1179.7127723693848P{œ…‚(:‘ ÔêëÔ²†ôÌ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚172.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.544000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àVì6}†‚(:ò ÔêëÔ’å‰Î) disk./.usagePercent‚79.17308511689345 disk.sda1.out‚0$ disk.dm-0.out‚100167.96923828125 disk.dm-1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 11933.2109375 disk.sda3.out‚ 174.9609375 disk.sda.in‚0 disk.sda.out‚ 174.9609375 disk.sda1.in‚0 disk.drbd1260.in‚0$ memory_percent‚96.20223045349121' proc.memory.availableMB‚ 622.2265625* proc.memory.percent‚18.663406372070312$ disk./.usageGB‚1179.736183166504 disk.dm-1.out‚ 174.9609375 disk.drbd1260.out‚ 0.71484375# disk.nvme0c0n1.in‚ 11933.2109375) disk.nvme0c0n1.out‚100167.96923828125 cpu‚25.36853349343352 network.recv‚ 1.664864e+06 disk.dm-0.in‚ 11933.2109375 disk.sda2.out‚0' disk.nvme0n1.out‚100167.96923828125 disk.sda3.in‚0 proc.memory.rssMB‚ 3057.8125 proc.cpu.threads‚17 network.sent‚ 8.953484e+06 «é‡‚(:… ãêëÔºœôÍ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï -ˆ‚(:ô ãêëÔÛ·ŽÑ disk.sda3.out‚ 175.80859375 disk.sda.in‚0 disk.sda.out‚ 175.80859375% disk./.usageGB‚1179.7127723693848 disk.dm-0.in‚ 11933.2109375 disk.sda1.out‚0 proc.memory.rssMB‚ 3057.8125* proc.memory.percent‚18.663406372070312 cpu‚25.381919310194537) disk./.usagePercent‚79.17151399865581 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚96.19927406311035' proc.memory.availableMB‚ 622.7109375 proc.cpu.threads‚17 network.sent‚ 8.959729e+06 network.recv‚ 1.669676e+06 disk.dm-1.out‚ 175.80859375 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.71875) disk.nvme0c0n1.out‚100571.45751953125' disk.nvme0n1.out‚100571.45751953125$ disk.dm-0.out‚100571.45751953125 disk.dm-1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 11933.2109375! disk.nvme0n1.in‚ 11933.2109375C zщ‚(:’ òêëÔ×ÖèÊ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Ï‚Š‚(:÷ òêëÔÎûÎ disk.dm-1.in‚0* proc.memory.percent‚18.663406372070312) disk./.usagePercent‚79.16043808869443) disk.nvme0c0n1.out‚100869.08642578125 disk.sda.out‚ 177.01953125 disk.dm-1.out‚ 177.01953125 disk.drbd1260.in‚0 disk.drbd1260.out‚0.71875' proc.memory.availableMB‚ 622.7109375 proc.cpu.threads‚17 network.recv‚ 1.673016e+06 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.19927406311035 network.sent‚ 8.964705e+06' disk.nvme0n1.out‚100869.08642578125 disk.sda.in‚0 disk.dm-0.in‚12056.93359375 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 3057.8125 cpu‚25.394555331801214% disk./.usageGB‚1179.5477333068848$ disk.nvme0c0n1.in‚12056.93359375" disk.nvme0n1.in‚12056.93359375 disk.sda3.in‚0 disk.sda3.out‚ 177.01953125$ disk.dm-0.out‚100869.08642578125NrðÑ2‹- _wandbruntime‚5071  _runtime‚5071³›sW*ŒÑT cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚341552052303872 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚337206668088832 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4345189772800 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20712 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚75859574 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚615 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14401142784 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚75859574 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4156016242176 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚299662033867223 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167729705 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683038720 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701041664 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 170502144 _step‚294 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚67534858 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚91870133 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767680 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038720 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20710 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚305633540895232 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚75859572 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32005 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚45452218 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚295522236028656 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106804 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67535081 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32248 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚22082861 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416517 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167750417 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚305633540895232 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚309788874098688 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚309789575140352 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067444 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4155333203456 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚91870133 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 trainer/global_step‚3626 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚309789575140352 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32005 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚45452220 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚570 # _timestamp‚1.78854027612104e+09 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚75859572 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1268 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚22082640 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728389 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91890843 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4140478906011 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚299662033867223 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14401142784 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167729705 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683038720 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701041664 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 170502144 _step‚294 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚67534858 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚91870133 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189767680 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 683038720 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20710 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚305633540895232 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚75859572 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32005 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚45452218 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚295522236028656 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106804 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67535081 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32248 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚22082861 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13684416517 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167750417 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚305633540895232 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0¦¢÷û82Ž- _wandbruntime‚5072  _runtime‚5072;Ñc)ÝR SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚75875048 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 167774707 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91899657 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67549816 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737856 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚309844325898240 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚305687819805184 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 167753995 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885402773451552e+09 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767680 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067444 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚309844325898240 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4140286049451 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167774707 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683038720 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91878947 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4345850125824 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106804 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4156506093056 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚341608809708544 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91878947 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737856 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚337262959582720 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701041664 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚222 _step‚295ž/Úæh)âR VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 409073152 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚337262935642624 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚75875048 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32005 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 4155823054336 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 167774707 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91899657 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67549816 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 14135737856 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚309844325898240 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚305687819805184 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 167753995 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 % _timestamp‚1.7885402773451552e+09 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1189767680 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 681067444 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚309844325898240 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4140286049451 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167774707 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 683038720 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91878947 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4345850125824 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106804 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4156506093056 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚341608809708544 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91878947 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135737856 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚337262959582720 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 701041664 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚222 _step‚295 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚75875048 B&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2253299057483673 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 particle/accuracy_1_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9920440912246704 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.26631543040275574 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.34897470474243164 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002186928642913699 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4528590142726898 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.30933308601379395 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.523080050945282 -jet/accuracy_4_of_4‚0.0021919880528002977 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.35257408022880554 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.42870375514030457 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9597299695014954 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.990101158618927 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34476083517074585 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4068452715873718 *jet/accuracy_1_of_3‚0.8261330127716064 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3686637580394745 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 jet/accuracy_1_of_7‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9625735878944397 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3556201756000519 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8475992679595947 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2083333432674408 jet/accuracy_4_of_7‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18876907229423523 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3795962929725647 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4675925672054291 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9832128286361694 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9500992298126221 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.974173367023468 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5282266736030579 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_6‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42903003096580505 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3150521218776703 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 6 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.998236358165741 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ) Purity/1FRt*SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.30977293848991394 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9330356121063232 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18851351737976074 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23603099584579468 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3888500928878784 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5724816918373108 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 _step‚296 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.46930992603302 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22795552015304565 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0312500037252903 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2577756345272064 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0026455027982592583 +particle/specificity‚0.9566818475723267 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32269173860549927 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5003589987754822 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 _runtime‚5090.053226252 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5559566020965576 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4096830189228058 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5434209704399109 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666269302368 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9920262694358826 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3664630055427551 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13566939532756805 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.019013579934835434 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16787037253379822 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.24533289670944214 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22218497097492218 particle/accuracy_2_of_4‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2736842632293701 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3304598033428192 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24541978538036346 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4184111952781677 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.43662136793136597 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2847222089767456 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2312677800655365 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8006352782249451 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.256908655166626 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14819836616516113 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.43127185106277466 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17908793687820435 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08942296355962753 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4232456386089325 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3076252341270447 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3911469876766205 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 +jet/accuracy_3_of_4‚0.05030098184943199 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9716607928276062 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13672035932540894 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4036482274532318 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.27385056018829346 +jet/accuracy_3_of_5‚0.08870328217744827 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.8955027461051941 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.45997533202171326 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8713769316673279 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5337877869606018 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9623960852622986 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20601850748062134 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.6875 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.976031482219696 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6786403059959412 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004827577155083418 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 particle/accuracy_6_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3373934328556061 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36666664481163025 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22426444292068481 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9719294905662537 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.41395696997642517 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.943173885345459 5 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20689657330513 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35185185074806213 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.6366666555404663 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8500000238418579 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3250744044780731 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20214863121509552 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16787037253379822 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.01682780124247074 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2281746119260788 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 'particle/f_score‚0.3457038700580597 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9664732813835144 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.699999988079071 jet/accuracy_4_of_6‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8871527910232544 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14082886278629303 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9625785946846008 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.018558142706751823 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.25809648633003235 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16972990334033966 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9870758056640625 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18454331159591675 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.21604032814502716 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16822755336761475 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9974206686019897 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.6875 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.41986218094825745 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00954891461879015 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.43433985114097595 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2591779828071594 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21666212379932404 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_6‚0.057692307978868484 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24996840953826904 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24177055060863495 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15935368835926056 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8833333849906921 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.42502111196517944 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.257602334022522 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1657974272966385 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5406745672225952 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.49327996373176575 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2205991894006729 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3982524871826172 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34973546862602234 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.37795573472976685 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_5‚1 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3624545931816101 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878788113594055 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4019138514995575 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013095238246023655 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9920440912246704 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2763157784938812 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2862069010734558 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9920760989189148 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8698830604553223 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 +jet/accuracy_2_of_2‚0.09571389853954315 particle/accuracy_1_of_2‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9791130423545837 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007097164168953896 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.010707986541092396 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8841608166694641 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2925926148891449 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30685126781463623 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13333332538604736 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.955365777015686 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1769547462463379 particle/accuracy_4_of_4‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3775317966938019 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22614942491054535 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7091970443725586 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3322380781173706 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.21935194730758667 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49106019735336304 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9846490621566772 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9785156846046448 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3643074631690979 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2397477775812149 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9758821129798889 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08607830852270126 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.633228600025177 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7083333134651184 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47816094756126404 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3659158945083618 +jet/accuracy_4_of_5‚0.01708909310400486 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2453254610300064 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16457249224185944 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3621527850627899 particle/accuracy_1_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.38457486033439636 particle/accuracy_4_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04297074303030968 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9898374080657959 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9618726372718811 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3991118371486664 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3749329149723053 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4423076808452606 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38745251297950745 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24800017476081848 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9480785727500916 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5170139074325562 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403507739305496 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5635116696357727 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9809837937355042 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.399382084608078 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9844961166381836 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3624393045902252 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9395929574966431 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.016945121809840202 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3888500928878784 particle/accuracy_3_of_7‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9708674550056458 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02236943505704403 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36255356669425964 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04125615581870079 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4150208830833435 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34140291810035706 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833432674408 particle/accuracy_5_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08835168927907944 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15803711116313934 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3488215208053589 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.2003971834±ÓTÊ20137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878789007663727 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 )loss/val_total_loss‚50.40732394871066 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9786303043365479 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_2_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12220270186662674 5 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.98759925365448 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚3626 $ _timestamp‚1.788540294738455e+09 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07394768297672272 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.37344110012054443 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1924242377281189 particle/accuracy_4_of_5‚1 jet/accuracy_6_of_7‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468254029750824 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.23858363926410675 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30851033329963684 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9866121411323547 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9585070013999939 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.32684698700904846 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/1FRt0SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004617604892700911 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5273162722587585 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9681615233421326 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_5_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.31285399198532104 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 epoch‚2 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5130952596664429 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.37614771723747253 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18525071442127228 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2130209505558014 jet/accuracy_7_of_7‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012395673431456089 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_3_of_4‚1 + Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3125 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24246032536029816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20318052172660828 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5554275512695312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0464596152305603 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5586113929748535 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7validation_average_jet_accuracy‚0.39874133467674255 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272`çb7(k•í¥ 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5282266736030579 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_6‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42903003096580505 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3150521218776703 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 6 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.998236358165741 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ) Purity/1FRt*SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.30977293848991394 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9330356121063232 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.18851351737976074 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23603099584579468 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3888500928878784 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5724816918373108 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 _step‚296 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.46930992603302 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22795552015304565 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0312500037252903 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2577756345272064 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0026455027982592583 +particle/specificity‚0.9566818475723267 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32269173860549927 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5003589987754822 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 _runtime‚5090.053226252 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5559566020965576 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4096830189228058 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5434209704399109 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666269302368 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9920262694358826 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3664630055427551 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13566939532756805 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.019013579934835434 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16787037253379822 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.24533289670944214 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22218497097492218 particle/accuracy_2_of_4‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2736842632293701 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3304598033428192 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24541978538036346 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4184111952781677 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.43662136793136597 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2847222089767456 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2312677800655365 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8006352782249451 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.256908655166626 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14819836616516113 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.43127185106277466 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17908793687820435 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08942296355962753 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4232456386089325 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3076252341270447 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3911469876766205 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 +jet/accuracy_3_of_4‚0.05030098184943199 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9716607928276062 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13672035932540894 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4036482274532318 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.27385056018829346 +jet/accuracy_3_of_5‚0.08870328217744827 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.8955027461051941 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.45997533202171326 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8713769316673279 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5337877869606018 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9623960852622986 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20601850748062134 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.6875 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.976031482219696 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6786403059959412 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004827577155083418 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 particle/accuracy_6_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3373934328556061 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36666664481163025 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22426444292068481 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9719294905662537 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.41395696997642517 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.943173885345459 5 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20689657330513 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.35185185074806213 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.6366666555404663 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8500000238418579 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3250744044780731 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20214863121509552 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16787037253379822 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.01682780124247074 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2281746119260788 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 'particle/f_score‚0.3457038700580597 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9664732813835144 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.699999988079071 jet/accuracy_4_of_6‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8871527910232544 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14082886278629303 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9625785946846008 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.018558142706751823 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,particle/sensitivity‚0.25809648633003235 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16972990334033966 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9870758056640625 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18454331159591675 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.21604032814502716 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16822755336761475 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9974206686019897 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.6875 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.41986218094825745 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.00954891461879015 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.43433985114097595 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2591779828071594 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21666212379932404 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_6‚0.057692307978868484 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24996840953826904 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24177055060863495 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15935368835926056 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8833333849906921 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.42502111196517944 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.257602334022522 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754689216614 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1657974272966385 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5406745672225952 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.49327996373176575 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.10000000149011612 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2205991894006729 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3982524871826172 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34973546862602234 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.37795573472976685 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_5‚1 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3624545931816101 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878788113594055 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4019138514995575 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013095238246023655 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9920440912246704 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2763157784938812 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2862069010734558 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9920760989189148 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8698830604553223 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 +jet/accuracy_2_of_2‚0.09571389853954315 particle/accuracy_1_of_2‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9791130423545837 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.007097164168953896 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.010707986541092396 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8841608166694641 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2925926148891449 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30685126781463623 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13333332538604736 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.955365777015686 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1769547462463379 particle/accuracy_4_of_4‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3775317966938019 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22614942491054535 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7091970443725586 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3322380781173706 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.21935194730758667 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49106019735336304 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9846490621566772 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 particle/accuracy_2_of_7‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9785156846046448 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3643074631690979 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2397477775812149 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9758821129798889 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08607830852270126 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.633228600025177 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7083333134651184 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47816094756126404 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3659158945083618 +jet/accuracy_4_of_5‚0.01708909310400486 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1458333432674408 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2453254610300064 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16457249224185944 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3621527850627899 particle/accuracy_1_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 *jet/accuracy_1_of_6‚0.9615384340286255 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.38457486033439636 particle/accuracy_4_of_6‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04297074303030968 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9898374080657959 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9618726372718811 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3991118371486664 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3749329149723053 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4423076808452606 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38745251297950745 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24800017476081848 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13120567798614502 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9480785727500916 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5170139074325562 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403507739305496 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5635116696357727 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9809837937355042 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.399382084608078 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9844961166381836 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3624393045902252 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9395929574966431 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.016945121809840202 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3888500928878784 particle/accuracy_3_of_7‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9708674550056458 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02236943505704403 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36255356669425964 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04125615581870079 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4150208830833435 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34140291810035706 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833432674408 particle/accuracy_5_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08835168927907944 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15803711116313934 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3488215208053589 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878789007663727 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 )loss/val_total_loss‚50.40732394871066 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9786303043365479 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_2_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_pur–õ¨9Ì'ity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12220270186662674 5 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.98759925365448 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚3626 $ _timestamp‚1.788540294738455e+09 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07394768297672272 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.37344110012054443 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1924242377281189 particle/accuracy_4_of_5‚1 jet/accuracy_6_of_7‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1468254029750824 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.23858363926410675 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30851033329963684 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9866121411323547 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9585070013999939 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.32684698700904846 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/1FRt0SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004617604892700911 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5273162722587585 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9681615233421326 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_5_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.31285399198532104 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 epoch‚2 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5130952596664429 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.37614771723747253 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18525071442127228 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2130209505558014 jet/accuracy_7_of_7‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012395673431456089 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_3_of_4‚1 + Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3125 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1458333432674408 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24246032536029816 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20318052172660828 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5554275512695312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0464596152305603 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5586113929748535 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7validation_average_jet_accuracy‚0.39874133467674255 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04245283082127571 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3881944417953491 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9941020011901855 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2253299057483673 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 particle/accuracy_1_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9920440912246704 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.26631543040275574 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.34897470474243164 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002186928642913699 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_7‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4528590142726898 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.30933308601379395 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.523080050945282 -jet/accuracy_4_of_4‚0.0021919880528002977 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_7‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.35257408022880554 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.42870375514030457 particle/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9597299695014954 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.990101158618927 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34476083517074585 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4068452715873718 *jet/accuracy_1_of_3‚0.8261330127716064 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3686637580394745 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 jet/accuracy_1_of_7‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9625735878944397 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3556201756000519 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8475992679595947 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2083333432674408 jet/accuracy_4_of_7‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18876907229423523 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3795962929725647 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4675925672054291 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9832128286361694 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9500992298126221 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.974173367023468 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1¨èœ]ž–‚(:“ ëëÔÙ³´Ë gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚168.478' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.695600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t€P↗‚(:û ëëÔ†ÁÙÌ% disk./.usageGB‚1179.7074165344238 disk.drbd1260.out‚ 0.72265625" disk.nvme0n1.in‚12056.93359375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1322.5546875 disk.dm-0.in‚12056.93359375$ disk.dm-0.out‚102182.69873046875 disk.dm-1.out‚ 178.0390625) disk.nvme0c0n1.out‚102182.69873046875 disk.sda3.in‚0$ memory_percent‚91.92776679992676" proc.memory.rssMB‚ 3053.1171875 cpu‚25.410432039436337 network.sent‚ 9.078229e+06 network.recv‚ 1.682997e+06 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 178.0390625 proc.cpu.threads‚17) disk./.usagePercent‚79.17115456407774 disk.dm-1.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚12056.93359375' disk.nvme0n1.out‚102182.69873046875 disk.sda3.out‚ 178.0390625* proc.memory.percent‚18.634748458862305åÍLŽž˜‚(:“ ŸëëÔ ËÆÊ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žx‰Fx™‚(:í ŸëëÔ„ÒßÍ" disk.nvme0n1.in‚12056.93359375 disk.sda.in‚0% disk./.usageGB‚1179.6840286254883$ disk.nvme0c0n1.in‚12056.93359375$ disk.nvme0n1.out‚102321.95703125 disk.sda3.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 1322.8125" proc.memory.rssMB‚ 3053.1171875 disk.drbd1260.in‚0) disk./.usagePercent‚79.16958498188532 disk.dm-1.in‚0 disk.sda3.out‚ 178.9609375 disk.sda.out‚ 178.9609375 disk.sda1.out‚0* proc.memory.percent‚18.634748458862305 cpu‚25.40961638792355 disk.dm-1.out‚ 178.9609375 disk.drbd1260.out‚ 0.7265625& disk.nvme0c0n1.out‚102321.95703125$ memory_percent‚91.92619323730469 proc.cpu.threads‚17 network.sent‚ 9.202498e+06 network.recv‚ 1.689512e+06 disk.dm-0.in‚12056.93359375! disk.dm-0.out‚102321.95703125 disk.sda2.in‚0 disk.sda2.out‚0Çà 2š- _wandbruntime‚5122  _runtime‚5122 0lÇ›Á _runtime‚5122.829667589 _step‚297 & lr-AdamW/pg1‚0.0014848736350295828 & lr-AdamW/pg2‚0.0014848736350295828 trainer/global_step‚3649 % _timestamp‚1.7885403275186768e+091±m<ÌœÆ _step‚297 & lr-AdamW/pg1‚0.0014848736350295828 & lr-AdamW/pg2‚0.0014848736350295828 trainer/global_step‚3649 % _timestamp‚1.7885403275186768e+09 _runtime‚5122.829667589©DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312376389371392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308188831484416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92538992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76519758 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169085094 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚3649 _runtime‚5122.831509912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339979379339776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4186647915520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312375461396992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92538992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68135331 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4377959632896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344357583306240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚298 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚308188813481472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344357338972672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187557886976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22283813 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297966588581200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76519756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45851516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312375461396992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4170995690888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308188831484416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312376389371392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76519756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268îb‡ÖÅ#ŸðO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308188813481472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22283177 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312376389371392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76519756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4378180026368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92565336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297966605620560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312376389371392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308188831484416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92538992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76519758 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169085094 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚3649 _runtime‚5122.831509912 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339979379339776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4186647915520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312375461396992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92538992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68135331 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4377959632896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344357583306240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 _step‚298 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚308188813481472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344357338972672 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187557886976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22283813 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297966588581200 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76519756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.free´…)1d‚45851516 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312375461396992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4170995690888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚308188831484416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22288959 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68153185 _runtime‚5124.064845766 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4188679360512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885403287535777e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92590193 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308270450570240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76539916 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302218644154482 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76539918 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169103739 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92590193 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312459129930752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45863581 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340069605044224 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68152540 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76539916 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92563823 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4172112964166 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169130111 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344448715328512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312458201943040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45863583 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298045606175744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4187769375744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308270432567296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4379110284288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308270450570240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312459129930752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22289602 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302217719139910 trainer/global_step‚3649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169103739 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298045623215104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169130111/A‰a'¢ûM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68153185 _runtime‚5124.064845766 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4188679360512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885403287535777e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92590193 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308270450570240 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76539916 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302218644154482 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76539918 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169103739 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92590193 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312459129930752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45863581 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340069605044224 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68152540 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76539916 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92563823 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4172112964166 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169130111 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344448715328512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312458201943040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45863583 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298045606175744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚643 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4187769375744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚308270432567296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4379110284288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚308270450570240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312459129930752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22289602 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302217719139910 trainer/global_step‚3649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169103739 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚298045623215104 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169130111 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4379347441664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4188679360512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76539918 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344448976425984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92563823 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261097472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22288959 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572«ìøG'2£- _wandbruntime‚5124  _runtime‚5124ž5H¤ 2loss/SRqqt1/assignment_loss‚2.1375282848110566 /loss/FBt4/detection_loss‚0.9723546504974365 0loss/FBt1/assignment_loss‚0.6451474294547498 0loss/FBt3/assignment_loss‚0.5774256511657667 0loss/SRqqt1/detection_loss‚2.819131074043421 1loss/SRqqt2/detection_loss‚2.6372041634537955 .loss/FRt2/detection_loss‚2.035558738867226 0loss/FRt3/assignment_loss‚3.7479189425706863 .loss/FBt3/detection_loss‚2.042372948179642 /loss/FRt3/detection_loss‚2.4026020832020096 .loss/FRt4/detection_loss‚1.723873079850756 % _timestamp‚1.7885403287558475e+09 0loss/FRt1/assignment_loss‚3.8244697213172913 /loss/FRt4/assignment_loss‚1.570865525253888 2loss/SRqqt2/assignment_loss‚1.8481616353700785 2loss/SRqqt3/assignment_loss‚2.1016277014277875 /loss/FRt1/detection_loss‚2.4157105730846524 /loss/FBt1/detection_loss‚2.6773473445106957 _runtime‚5124.066604351 1loss/SRqqt4/assignment_loss‚1.028880553801769 1loss/FBt2/assignment_loss‚0.19649784692628777 1loss/SRqqt4/detection_loss‚0.9965083099387843 trainer/global_step‚3649 loss/FBt4/assignment_loss‚0 1loss/SRqqt3/detection_loss‚2.4758156333118677 _step‚300 /loss/FRt2/assignment_loss‚2.589287292698155 /loss/FBt2/detection_loss‚1.0467610311712705 %loss/total_loss‚44.51305021490907 epoch‚3À€E¾M¥Ç /loss/FRt2/assignment_loss‚2.589287292698155 /loss/FBt2/detection_loss‚1.0467610311712705 %loss/total_loss‚44.51305021490907 epoch‚3 2loss/SRqqt1/assignment_loss‚2.1375282848110566 /loss/FBt4/detection_loss‚0.9723546504974365 0loss/FBt1/assignment_loss‚0.6451474294547498 0loss/FBt3/assignment_loss‚0.5774256511657667 0loss/SRqqt1/detection_loss‚2.819131074043421 1loss/SRqqt2/detection_loss‚2.6372041634537955 .loss/FRt2/detection_loss‚2.035558738867226 0loss/FRt3/assignment_loss‚3.7479189425706863 .loss/FBt3/detection_loss‚2.042372948179642 /loss/FRt3/detection_loss‚2.4026020832020096 .loss/FRt4/detection_loss‚1.723873079850756 % _timestamp‚1.7885403287558475e+09 0loss/FRt1/assignment_loss‚3.8244697213172913 /loss/FRt4/assignment_loss‚1.570865525253888 2loss/SRqqt2/assignment_loss‚1.8481616353700785 2loss/SRqqt3/assignment_loss‚2.1016277014277875 /loss/FRt1/detection_loss‚2.4157105730846524 /loss/FBt1/detection_loss‚2.6773473445106957 _runtime‚5124.066604351 1loss/SRqqt4/assignment_loss‚1.028880553801769 1loss/FBt2/assignment_loss‚0.19649784692628777 1loss/SRqqt4/detection_loss‚0.9965083099387843 trainer/global_step‚3649 loss/FBt4/assignment_loss‚0 1loss/SRqqt3/detection_loss‚2.4758156333118677 _step‚300¬T°@“¦‚(:ˆ ®ëëÔžêÇÎ gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚124.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N%ù|§‚(:ñ ®ëëÔšÖ«Ñ) disk./.usagePercent‚79.17098457507501% disk./.usageGB‚1179.7048835754395 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 3053.1171875 proc.cpu.threads‚17 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚12202.76953125" disk.nvme0n1.in‚12202.76953125 disk.sda3.out‚ 179.6171875 disk.sda.out‚ 179.6171875! disk.dm-0.out‚103367.30859375$ memory_percent‚91.92788600921631* proc.memory.percent‚18.634748458862305 network.sent‚ 9.207531e+06 disk.sda2.out‚0$ disk.nvme0n1.out‚103367.30859375 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚12202.76953125) proc.memory.availableMB‚ 1322.53515625 cpu‚25.40868292203276 network.recv‚ 1.693103e+06 disk.drbd1260.out‚ 0.7265625& disk.nvme0c0n1.out‚103367.30859375 disk.sda3.in‚0 disk.dm-1.out‚ 179.6171875oM“¨‚(:ˆ ½ëëÔÕùÝË gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª3ˆýŠ©‚(:ÿ ½ëëÔÆûÿÏ$ memory_percent‚91.92640781402588 network.recv‚ 1.698693e+06% disk./.usageGB‚1179.7048645019531 disk.dm-0.in‚12202.76953125 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.634748458862305 network.sent‚ 9.317265e+06 disk.sda2.in‚0" disk.nvme0n1.in‚12202.76953125 disk.sda1.in‚0) proc.memory.availableMB‚ 1322.77734375 proc.cpu.threads‚17) disk./.usagePercent‚79.17098329503733$ disk.dm-0.out‚103602.70263671875 disk.dm-1.out‚ 180.05859375 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚103602.70263671875 disk.sda.in‚0" proc.memory.rssMB‚ 3053.1171875 cpu‚25.407949511754545 disk.drbd1260.out‚ 0.73046875$ disk.nvme0c0n1.in‚12202.76953125' disk.nvme0n1.out‚103602.70263671875 disk.sda3.out‚ 180.05859375 disk.sda.out‚ 180.05859375ÔۇȞª‚(:“ ÌëëÔË×¼Ê gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚143.688' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.737599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:`Œr‚«‚(:÷ ÌëëÔ¬ƒ‚Î disk.dm-1.in‚0) disk.nvme0c0n1.out‚103920.65966796875' disk.nvme0n1.out‚103920.65966796875 disk.sda3.in‚0 disk.sda.out‚ 180.390625* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17 network.sent‚ 9.32125e+06 network.recv‚ 1.702675e+06 disk.dm-0.in‚12202.76953125 disk.dm-1.out‚ 180.390625 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 180.390625 disk.sda.in‚0$ disk./.usageGB‚1179.597023010254$ disk.dm-0.out‚103920.65966796875 disk.sda2.in‚0" disk.nvme0n1.in‚12202.76953125 disk.sda1.out‚0$ memory_percent‚91.92640781402588) proc.memory.availableMB‚ 1322.77734375" proc.memory.rssMB‚ 3053.1171875$ disk.nvme0c0n1.in‚12202.76953125) disk./.usagePercent‚79.16374596204436 disk.drbd1260.out‚ 0.73046875 disk.sda1.in‚0 cpu‚25.407034593879647.› “¬‚(:ˆ ÛëëÔ²í Ê gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/W5°s­‚(:è ÛëëÔðýÒË disk.sda2.out‚0$ disk.nvme0c0n1.in‚12202.76953125 disk.sda.out‚180.875$ memory_percent‚91.92793369293213 proc.cpu.threads‚17" disk.dm-0.out‚104180.111328125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚104180.111328125 disk.sda3.in‚0* proc.memory.percent‚18.634748458862305% disk./.usageGB‚1179.5970420837402 disk.dm-0.in‚12202.76953125' disk.nvme0c0n1.out‚104180.111328125 disk.sda.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3053.1171875 cpu‚25.406224653974398 network.sent‚ 9.327464e+06 disk.dm-1.out‚180.875 disk.drbd1260.out‚0.734375" disk.nvme0n1.in‚12202.76953125 disk.sda3.out‚180.875 disk.sda1.out‚0) proc.memory.availableMB‚ 1322.52734375 network.recv‚ 1.707333e+06) disk./.usagePercent‚79.16374724208204Ô’2®- _wandbruntime‚5182  _runtime‚5182§?øxï½ $ lr-AdamW/pg2‚0.00147692780339667 trainer/global_step‚3699 % _timestamp‚1.7885403869376316e+09 _runtime‚5182.248592689 _step‚301 $ lr-AdamW/pg1‚0.00147692780339667ú_`È°Â $ lr-AdamW/pg2‚0.00147692780339667 trainer/global_step‚3699 % _timestamp‚1.7885403869376316e+09 _runtime‚5182.248592689 _step‚301 $ lr-AdamW/pg1‚0.00147692780339667­—¤ ,2±- _wandbruntime‚5182  _runtime‚5182‘øÒÁ ²ìO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93810238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22570837 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316513431087616 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 118772Îwèæ8393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4243645312000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4242735340544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4436230401024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885403869387774e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4227780921000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69025000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312269767772672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312269785775616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4242735340544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348927135102464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93810238 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171337996 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4436446600192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306145266268760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77527758 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316512503113216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348926894963200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0:_G‹÷'³ñO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4243645312000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4242735340544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4436230401024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885403869387774e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4227780921000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69025000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312269767772672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312269785775616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4242735340544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348927135102464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93810238 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171337996 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4436446600192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306145266268760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77527758 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316512503113216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348926894963200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93810238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22570837 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚302 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316513431087616®Û6ká2´- _wandbruntime‚5183  _runtime‚5183+*çýü&µöM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301996485902944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171383013 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77547918 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69042808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171356641 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312351404861440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171356641 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93835095 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4244766785536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77547916 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4228898194486 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301996502942304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349018270401536 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316596171646976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚3699 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93808725 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46466831 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316595243659264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22576574 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306224476122218 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22575977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚597 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344580890266624 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171383013 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312351404861440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312351386858496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312351386858496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93835095 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599}óÒ¶ûM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312351386858496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93835095 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301996485902944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171383013 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77547918 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69042808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171356641 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312351404861440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171356641 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93835095 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4244766785536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77547916 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4228898194486 P@Deviceõô/StatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301996502942304 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349018270401536 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316596171646976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚3699 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93808725 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46466831 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316595243659264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22576574 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚306224476122218 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22575977 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚597 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 _step‚303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344580890266624 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171383013 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312351404861440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312351386858496¯¯Aò«2·- _wandbruntime‚5183  _runtime‚5183Cˆ+ŽE¸¿ /loss/SRqqt4/detection_loss‚1.20014939344198 /loss/FRt2/detection_loss‚2.5700314887093776 1loss/SRqqt1/assignment_loss‚1.668218720704317 1loss/SRqqt3/assignment_loss‚2.004233617048997 0loss/FBt1/assignment_loss‚0.4844681467882104 loss/FBt4/assignment_loss‚0 /loss/FRt1/detection_loss‚2.6880417948260025 /loss/FRt3/detection_loss‚2.3512699902198757 0loss/SRqqt3/detection_loss‚2.707672968506813 2loss/SRqqt2/assignment_loss‚1.7372933543390698 .loss/FBt1/detection_loss‚2.531819898973812 trainer/global_step‚3699 % _timestamp‚1.7885403884690175e+09 /loss/FBt2/detection_loss‚1.1988436732321017 0loss/FRt1/assignment_loss‚3.4761659485452316 /loss/FRt3/assignment_loss‚4.118500485564724 /loss/FRt4/assignment_loss‚2.210304657914745 2loss/SRqqt4/assignment_loss‚1.2221865459940717 0loss/FBt3/assignment_loss‚0.5840110362833286 %loss/total_loss‚45.21285403537557 _runtime‚5183.779816315 /loss/FRt2/assignment_loss‚3.138730401342565 epoch‚3 1loss/FBt2/assignment_loss‚0.31794079447087875 .loss/FBt3/detection_loss‚1.594286091901638 /loss/FBt4/detection_loss‚0.8755598068237305 _step‚304 /loss/FRt4/detection_loss‚1.9539870920021143 1loss/SRqqt1/detection_loss‚2.6796646692134716 1loss/SRqqt2/detection_loss‚1.8994734585285187 ¥…J¹Ä 0loss/FBt3/assignment_loss‚0.5840110362833286 %loss/total_loss‚45.21285403537557 _runtime‚5183.779816315 /loss/FRt2/assignment_loss‚3.138730401342565 epoch‚3 1loss/FBt2/assignment_loss‚0.31794079447087875 .loss/FBt3/detection_loss‚1.594286091901638 /loss/FBt4/detection_loss‚0.8755598068237305 _step‚304 /loss/FRt4/detection_loss‚1.9539870920021143 1loss/SRqqt1/detection_loss‚2.6796646692134716 1loss/SRqqt2/detection_loss‚1.8994734585285187 /loss/SRqqt4/detection_loss‚1.20014939344198 /loss/FRt2/detection_loss‚2.5700314887093776 1loss/SRqqt1/assignment_loss‚1.668218720704317 1loss/SRqqt3/assignment_loss‚2.004233617048997 0loss/FBt1/assignment_loss‚0.4844681467882104 loss/FBt4/assignment_loss‚0 /loss/FRt1/detection_loss‚2.6880417948260025 /loss/FRt3/detection_loss‚2.3512699902198757 0loss/SRqqt3/detection_loss‚2.707672968506813 2loss/SRqqt2/assignment_loss‚1.7372933543390698 .loss/FBt1/detection_loss‚2.531819898973812 trainer/global_step‚3699 % _timestamp‚1.7885403884690175e+09 /loss/FBt2/detection_loss‚1.1988436732321017 0loss/FRt1/assignment_loss‚3.4761659485452316 /loss/FRt3/assignment_loss‚4.118500485564724 /loss/FRt4/assignment_loss‚2.210304657914745 2loss/SRqqt4/assignment_loss‚1.2221865459940717°»óƒÚžº‚(:“ êëëÔ•§üÐ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.565' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.312999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!+J~»‚(:ó êëëÔÕÅîÒ) proc.memory.availableMB‚ 1322.70703125 disk.sda2.out‚0' disk.nvme0c0n1.out‚104608.435546875! disk.nvme0n1.in‚ 12331.1484375 disk.sda3.in‚0 disk.sda3.out‚ 181.41796875" disk.dm-0.out‚104608.435546875 disk.dm-1.in‚0 cpu‚25.405371929039998 network.sent‚ 9.332559e+06 disk.sda2.in‚0 disk.sda.out‚ 181.41796875 disk.sda1.out‚0" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17% disk./.usageGB‚1179.7626838684082 disk.drbd1260.out‚0.734375% disk.nvme0n1.out‚104608.435546875 disk.sda.in‚0$ memory_percent‚91.92683696746826 network.recv‚ 1.71084e+06) disk./.usagePercent‚79.17486360123381 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 12331.1484375 disk.sda1.in‚0 disk.dm-0.in‚ 12331.1484375 disk.dm-1.out‚ 181.41796875žâ.“¼‚(:ˆ ùëëÔ¿ˆúÌ gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.384' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒØB½‚(:ô ùëëÔ¬¥øÐ) disk./.usagePercent‚79.17486232119614! disk.nvme0n1.in‚ 12331.1484375' disk.nvme0n1.out‚104757.42041015625 disk.sda3.out‚ 182.015625 disk.sda1.out‚0$ memory_percent‚91.92681312561035* proc.memory.percent‚18.634748458862305 network.recv‚ 1.715517e+06 disk.sda2.out‚0 disk.sda1.in‚0 cpu‚25.404565646710555 proc.cpu.threads‚17% disk./.usageGB‚1179.7626647949219 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.7421875) disk.nvme0c0n1.out‚104757.42041015625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 182.015625 disk.dm-0.in‚ 12331.1484375$ disk.dm-0.out‚104757.42041015625 disk.dm-1.out‚ 182.015625 disk.sda2.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 12331.1484375( proc.memory.availableMB‚ 1322.7109375" proc.memory.rssMB‚ 3053.1171875 network.sent‚ 9.440211e+06" •Òž¾‚(:“ ˆìëÔƒÞ‡Ò gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.187' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.437400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00öÒ9€¿‚(:õ ˆìëÔδÖ( proc.memory.availableMB‚ 1322.9453125 cpu‚25.403883051099616) disk./.usagePercent‚79.17485848108313! disk.nvme0n1.in‚ 12331.1484375' disk.nvme0n1.out‚104872.41259765625 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.7421875* proc.memory.percent‚18.634748458862305 network.recv‚ 1.71956e+06$ disk./.usageGB‚1179.762607574463 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 182.5703125# disk.nvme0c0n1.in‚ 12331.1484375) disk.nvme0c0n1.out‚104872.41259765625" proc.memory.rssMB‚ 3053.1171875 network.sent‚ 9.447411e+06 disk.dm-0.in‚ 12331.1484375 disk.dm-1.in‚0 disk.dm-1.out‚ 182.5703125 proc.cpu.threads‚17 disk.sda3.out‚ 182.5703125 disk.sda1.out‚0$ disk.dm-0.out‚104872.41259765625 disk.sda2.in‚0 disk.drbd1260.in‚0$ memory_percent‚91.92538261413574+ kžÀ‚(:“ —ìëÔ¯¡ÕÊ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.938' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.187599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…»Å‚Á‚(:÷ —ìëÔÏ* proc.memory.percent‚18.634748458862305 network.recv‚ 1.724627e+06) disk./.usagePercent‚79.17360199610508$ disk.nvme0c0n1.in‚12331.15234375 disk.sda3.out‚ 183.0546875 disk.dm-0.in‚12331.15234375 disk.sda2.out‚0$ memory_percent‚96.33402824401855 network.sent‚ 9.451757e+06( disk.nvme0c0n1.out‚105185.4521484375" disk.nvme0n1.in‚12331.15234375 disk.dm-1.out‚ 183.0546875' proc.memory.availableMB‚ 600.6328125 cpu‚25.403043851594795% disk./.usageGB‚1179.7438850402832 disk.sda.in‚0 disk.sda1.in‚0# disk.dm-0.out‚105185.4521484375 disk.drbd1260.out‚ 0.74609375" proc.memory.rssMB‚ 3053.1171875 proc.cpu.threads‚17& disk.nvme0n1.out‚105185.4521484375 disk.sda3.in‚0 disk.sda.out‚ 183.0546875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0Š´äò2Â- _wandbruntime‚5246  _runtime‚5246øí ¨ÇÃÁ _runtime‚5246.538005356 _step‚305 & lr-AdamW/pg1‚0.0014688498092989117 & lr-AdamW/pg2‚0.0014688498092989117 trainer/global_step‚3749 % _timestamp‚1.7885404512270567e+09›—ÌÄÆ & lr-AdamW/pg1‚0.0014688498092989117 & lr-AdamW/pg2‚0.0014688498092989117 trainer/global_step‚3749 % _timestamp‚1.7885404512270567e+09 _runtime‚5246.538005356 _step‚305±quŽè2Å- _wandbruntime‚5246  _runtime‚5246{ÿ£ò'ÆìO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚310151098236704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4282750201104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 _runtime‚5246.539392547 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4299732737024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95055140 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320649544829440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353496663776256 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885404512282176e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47058018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22858186 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4299732737024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95055140 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320649544829440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353496663776256 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 @:DeviceStatsMonitor.on_train_batch_start/T !‹ºnum_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885404512282176e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47058018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22858186 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173635915 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 % _timestamp‚1.7885404522821598e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353588306680320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95053627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4283867474382 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78555916 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316432341149696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22863859 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349092175489024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4284775449594 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173635915 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310232158119098 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320733213363200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173609543 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78555918 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4299944225792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349092199429120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69933324 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4496107251200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310231233104526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95079997 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305947365630144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4299944225792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305947382669504 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316432359152640 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173635915 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 % _timestamp‚1.7885404522821598e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚353588306680320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95053627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4283867474382 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78555916 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316432341149696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22863859 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349092175489024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4284775449594 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173635915 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚310232158119098 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320733213363200 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173609543 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78555918 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4299944225792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349092199429120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69933324 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4496107251200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310231233104526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95079997 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305947365630144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144³Wý;Î2Ë- _wandbruntime‚5247  _runtime‚5247¬§UpEÌ¿ /loss/FRt2/assignment_loss‚2.697104790518361 % _timestamp‚1.7885404522849627e+09 _runtime‚ 5247.59648318 /loss/FRt4/detection_loss‚1.7758022019314388 0loss/SRqqt3/detection_loss‚2.555378468140312 /loss/FBt1/detection_loss‚2.6693738909328686 /loss/FBt4/detection_loss‚0.4525904655456543 %loss/total_loss‚43.95960186489198 1loss/SRqqt2/detection_loss‚2.1849092421708285 /loss/FBt3/detection_loss‚1.9475370636209846 epoch‚3 /loss/FRt1/assignment_loss‚3.693950039148331 2loss/SRqqt1/assignment_loss‚1.9222869996143424 2loss/SRqqt4/assignment_loss‚1.2701049122232488 loss/FBt4/assignment_loss‚0 -loss/FRt2/detection_loss‚2.48761257577327 trainer/global_step‚3749 _step‚308 2loss/SRqqt2/assignment_loss‚1.7967278532407902 1loss/SRqqt3/assignment_loss‚2.126496213933696 .loss/FRt3/detection_loss‚2.372364403360656 0loss/SRqqt1/detection_loss‚2.760951192482658 1loss/FBt1/assignment_loss‚0.21091453866307242 0loss/FBt3/assignment_loss‚0.4894947131435477 /loss/FRt1/detection_loss‚2.3618864797055723 /loss/FBt2/detection_loss‚1.2878411124164069 /loss/FRt3/assignment_loss‚4.096495052746364 0loss/FRt4/assignment_loss‚1.7556141245932806 0loss/FBt2/assignment_loss‚0.2266633523105285 1loss/SRqqt4/detection_loss‚0.8175021786757601©/¬‰JÍÄ % _timestamp‚1.7885404522849627e+09 _runtime‚ 5247.59648318 /loss/FRt4/detection_loss‚1.7758022019314388 0loss/SRqqt3/detection_loss‚2.555378468140312 /loss/FBt1/detection_loss‚2.6693738909328686 /loss/FBt4/detection_loss‚0.4525904655456543 %loss/total_loss‚43.95960186489198 1loss/SRqqt2/detection_loss‚2.1849092421708285 /loss/FBt3/detection_loss‚1.9475370636209846 epoch‚3 /loss/FRt1/assignment_loss‚3.693950039148331 2loss/SRqqt1/assignment_loss‚1.9222869996143424 2loss/SRqqt4/assignment_loss‚1.2701049122232488 loss/FBt4/assignment_loss‚0 -loss/FRt2/detection_loss‚2.48761257577327 trainer/global_step‚3749 _step‚308 2loss/SRqqt2/assignment_loss‚1.7967278532407902 1loss/SRqqt3/assignment_loss‚2.126496213933696 .loss/FRt3/detection_loss‚2.372364403360656 0loss/SRqqt1/detection_loss‚2.760951192482658 1loss/FBt1/assignment_loss‚0.21091453866307242 0loss/FBt3/assignment_loss‚0.4894947131435477 /loss/FRt1/detection_loss‚2.3618864797055723 /loss/FBt2/detection_loss‚1.2878411124164069 /loss/FRt3/assignment_loss‚4.096495052746364 0loss/FRt4/assignment_loss‚1.7556141245932806 0loss/FBt2/assignment_loss‚0.2266633523105285 1loss/SRqqt4/detection_loss‚0.8175021786757601 /loss/FRt2/assignment_loss‚2.697104790518361´˜Ñu‹“΂(:ˆ ¦ìëÔÙÄåÎ gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9962 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q*ÐåƒÏ‚(:ø ¦ìëÔ›ò¿Ð proc.cpu.threads‚17 network.sent‚ 9.456851e+06% disk./.usageGB‚1179.7683181762695 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚12368.59765625" disk.nvme0n1.in‚12368.59765625 disk.sda1.out‚0 disk.dm-0.in‚12368.59765625 disk.drbd1260.out‚ 0.74609375& disk.nvme0n1.out‚105502.2724609375 disk.sda3.in‚0 disk.sda3.out‚ 183.73046875# disk.dm-0.out‚105502.2724609375 disk.dm-1.in‚0 disk.dm-1.out‚ 183.73046875$ memory_percent‚96.33426666259766 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚105502.2724609375 disk.sda.in‚0 disk.sda.out‚ 183.73046875 disk.sda1.in‚0% proc.memory.availableMB‚ 600.59375 cpu‚25.402266514423385 network.recv‚ 1.728257e+06) disk./.usagePercent‚79.17524172436188" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305“·Ÿ“Ђ(:ˆ µìëÔÛÚçÐ gpu.0.gpu‚21.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚160.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øw6qÑ‚(:æ µìëÔñò™Ø network.sent‚ 9.566437e+06 network.recv‚ 1.733665e+06' disk./.usagePercent‚79.179429495607% disk./.usageGB‚1179.8307189941406 disk.sda.in‚0 disk.sda.out‚ 184.34375 disk.sda1.in‚0 disk.dm-1.out‚ 184.34375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚12505.94140625" disk.nvme0n1.in‚12505.94140625% disk.nvme0n1.out‚105992.525390625 disk.sda3.in‚0 disk.sda3.out‚ 184.34375 disk.dm-0.in‚12505.94140625" disk.dm-0.out‚105992.525390625 disk.drbd1260.out‚0.75# memory_percent‚96.3327169418335" proc.memory.rssMB‚ 3053.1171875 cpu‚25.401431500276065 disk.dm-1.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚105992.525390625( proc.memory.availableMB‚ 600.84765625* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚173Ð=è“Ò‚(:ˆ ÄìëÔÚǹÎ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GyB|Ó‚(:ñ ÄìëÔÒ³’Ð cpu‚25.400662207202107 network.recv‚ 1.737777e+06 disk.sda3.in‚0 disk.sda3.out‚ 184.65234375 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 184.65234375 disk.drbd1260.out‚0.75' proc.memory.availableMB‚ 600.6015625 network.sent‚ 9.570422e+06' disk.nvme0c0n1.out‚106199.419921875% disk.nvme0n1.out‚106199.419921875 disk.sda.out‚ 184.65234375 disk.sda2.out‚0* proc.memory.percent‚18.634748458862305) disk./.usagePercent‚79.18362392304799" disk.nvme0n1.in‚12505.94140625 disk.sda2.in‚0 disk.drbd1260.in‚0$ memory_percent‚96.33421897888184" proc.memory.rssMB‚ 3053.1171875 proc.cpu.threads‚17% disk./.usageGB‚1179.8932189941406$ disk.nvme0c0n1.in‚12505.94140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚12505.94140625" disk.dm-0.out‚106199.419921875¢Î"Ô‚(:’ ÓìëÔàßåÍ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚167.952' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.590399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏVrz‚Õ‚(:÷ ÓìëÔ›º¶Ñ$ memory_percent‚96.33424282073975" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17 network.sent‚ 9.57661e+06 network.recv‚ 1.742214e+06% disk./.usageGB‚1180.0182189941406 disk.sda.in‚0 disk.sda.out‚ 185.0703125 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚12505.94140625( proc.memory.availableMB‚ 600.59765625 cpu‚25.399992584734303" disk.nvme0n1.in‚12505.94140625& disk.nvme0n1.out‚106501.3642578125 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚12505.94140625# disk.dm-0.out‚106501.3642578125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.75390625) disk./.usagePercent‚79.19201277792997 disk.sda3.out‚ 185.0703125 disk.dm-1.out‚ 185.0703125( disk.nvme0c0n1.out‚106501.36425781250×Z2Ö- _wandbruntime‚5306  _runtime‚5306Â>ÚÅ׿ & lr-AdamW/pg1‚0.0014606413361781699 & lr-AdamW/pg2‚0.0014606413361781699 trainer/global_step‚3799 $ _timestamp‚1.788540510751018e+09 _runtime‚ 5306.06194643 _step‚309óÛzéÊØÄ & lr-AdamW/pg1‚0.0014606413361781699 & lr-AdamW/pg2‚0.0014606413361781699 trainer/global_step‚3799 $ _timestamp‚1.788540510751018e+09 _runtime‚ 5306.06194643 _step‚309µÕ½Zá2Ù- _wandbruntime‚5306  _runtime‚5306ÌÅ( ÚìO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320431694358016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314158780233376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175817454 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175843800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4553152744960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70805408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324787514520064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324786586545664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4338627456212 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚310 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70806036 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353513235006976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96273698 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354910190592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚324787514520064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79543756 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47661268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96273698 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47661266 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96300042 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309819227762800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚3799 % _timestamp‚1.7885405107521572e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324786586545664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79543758 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23144768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358066154775552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4552919768576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354910190592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4339535431216 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320431676355072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353513258947072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320431676355072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79543756SÞM÷'ÛñO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320431676355072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79543756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320431694358016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314158780233376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175817454 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175843800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4553152744960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70805408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324787514520064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324786586545664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4338627456212 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 _step‚310 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70806036 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353513235006976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96273698 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354910190592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚324787514520064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79543756 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47661268 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96273698 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47661266 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96300042 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309819227762800 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚3799 % _timestamp‚1.7885405107521572e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324786586545664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79543758 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23144768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358066154775552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4552919768576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354910190592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4339535431216 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚320431676355072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚353513258947072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¶™/‘,2Ü- _wandbruntime‚5307  _runtime‚53076OñÎü&ÝöM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320513313443840 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324869327091712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320513295440896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309898262396704 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47673331 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚5307.071845525 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339744729490 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314237990086834 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96324899 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96298529 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175888817 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320513313443840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358157548494336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4356941635584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353603460711424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70823261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47673333 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4356031650816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96298529 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79563918 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23149930 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175862445 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4356031650816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175862445 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96324899 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340652704702 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358157801203200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353603484651520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚309898245357344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4554316551680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175888817 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324870255079424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 _step‚311 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96298529 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79563918 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23149930 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175862445 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4356031650816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175862445 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96324899 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4340652704702 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358157801203200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353603484651520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚309898245357344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4554316551680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175888817 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚324870255079424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 _step‚311 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3799 ZFDeviceStatsMon@Ο¬itor.on_train_batch_end/allocated_bytes.large_pool.freed‚320513295440896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309898262396704 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47673331 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 _runtime‚5307.071845525 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339744729490 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314237990086834 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96324899 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96298529 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175888817 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320513313443840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358157548494336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4356941635584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353603460711424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70823261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47673333 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4356031650816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231·íXf2ß- _wandbruntime‚5307  _runtime‚5307pû2ÌGàÁ /loss/FRt1/assignment_loss‚3.378945735997932 .loss/FRt3/detection_loss‚2.439819407369312 1loss/SRqqt2/detection_loss‚2.6527210146188738 epoch‚3 1loss/SRqqt4/detection_loss‚0.8628769996506509 /loss/FBt2/detection_loss‚0.9788878711443099 trainer/global_step‚3799 0loss/SRqqt1/detection_loss‚3.218644593249668 % _timestamp‚1.7885405117624822e+09 /loss/FRt3/assignment_loss‚3.504707503963161 0loss/FRt4/assignment_loss‚1.8123105118672054 1loss/FBt2/assignment_loss‚0.32551888858865136 _step‚312 .loss/FRt2/detection_loss‚2.483820068259393 .loss/FBt3/detection_loss‚1.767992126253935 0loss/FBt1/assignment_loss‚0.5365589449720574 %loss/total_loss‚43.92409279439839 2loss/SRqqt3/assignment_loss‚2.4430359921285083 2loss/SRqqt4/assignment_loss‚1.1936748349177304 loss/FBt4/assignment_loss‚0 _runtime‚5307.073372885 0loss/SRqqt3/detection_loss‚2.804368657725198 /loss/FRt1/detection_loss‚2.3042162923618803 /loss/FRt2/assignment_loss‚2.865367837490574 2loss/SRqqt1/assignment_loss‚1.7876501977443695 1loss/SRqqt2/assignment_loss‚1.342348912730813 /loss/FRt4/detection_loss‚1.8647246011843284 0loss/FBt3/assignment_loss‚0.5624325027390393 /loss/FBt1/detection_loss‚2.3061560288719507 0loss/FBt4/detection_loss‚0.48731327056884766á²ò¾LáÆ /loss/FRt1/detection_loss‚2.3042162923618803 /loss/FRt2/assignment_loss‚2.865367837490574 2loss/SRqqt1/assignment_loss‚1.7876501977443695 1loss/SRqqt2/assignment_loss‚1.342348912730813 /loss/FRt4/detection_loss‚1.8647246011843284 0loss/FBt3/assignment_loss‚0.5624325027390393 /loss/FBt1/detection_loss‚2.3061560288719507 0loss/FBt4/detection_loss‚0.48731327056884766 /loss/FRt1/assignment_loss‚3.378945735997932 .loss/FRt3/detection_loss‚2.439819407369312 1loss/SRqqt2/detection_loss‚2.6527210146188738 epoch‚3 1loss/SRqqt4/detection_loss‚0.8628769996506509 /loss/FBt2/detection_loss‚0.9788878711443099 trainer/global_step‚3799 0loss/SRqqt1/detection_loss‚3.218644593249668 % _timestamp‚1.7885405117624822e+09 /loss/FRt3/assignment_loss‚3.504707503963161 0loss/FRt4/assignment_loss‚1.8123105118672054 1loss/FBt2/assignment_loss‚0.32551888858865136 _step‚312 .loss/FRt2/detection_loss‚2.483820068259393 .loss/FBt3/detection_loss‚1.767992126253935 0loss/FBt1/assignment_loss‚0.5365589449720574 %loss/total_loss‚43.92409279439839 2loss/SRqqt3/assignment_loss‚2.4430359921285083 2loss/SRqqt4/assignment_loss‚1.1936748349177304 loss/FBt4/assignment_loss‚0 _runtime‚5307.073372885 0loss/SRqqt3/detection_loss‚2.804368657725198¸Ô rÿ…â‚(:ú âìëÔèäÚÎ network.recv‚ 1.745678e+06 disk.drbd1260.out‚ 0.75390625& disk.nvme0n1.out‚106678.8369140625 disk.dm-1.in‚0 cpu‚25.399132673754128 proc.cpu.threads‚17 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚106678.8369140625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 185.84765625 disk.sda1.out‚0 disk.dm-0.in‚12505.94140625$ memory_percent‚96.33307456970215 disk.sda2.out‚0$ disk.nvme0c0n1.in‚12505.94140625" disk.nvme0n1.in‚12505.94140625 disk.sda3.out‚ 185.84765625 disk.sda1.in‚0' proc.memory.availableMB‚ 600.7890625" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 network.sent‚ 9.594737e+06) disk./.usagePercent‚79.19040914673549% disk./.usageGB‚1179.9943237304688 disk.sda2.in‚0# disk.dm-0.out‚106678.8369140625 disk.dm-1.out‚ 185.84765625õ³1”žã‚(:“ âìëÔàÏÎ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.591' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.918200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03šòæ“ä‚(:ˆ ñìëÔݲ‹Ì gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œˆM«zå‚(:ï ñìëÔ†ºÏÏ! disk.nvme0c0n1.in‚ 12650.46875 disk.nvme0n1.in‚ 12650.46875 network.sent‚ 9.689388e+06) disk./.usagePercent‚79.20725521051111$ disk.dm-0.out‚106948.36767578125 disk.dm-1.out‚ 186.1640625) disk.nvme0c0n1.out‚106948.36767578125' disk.nvme0n1.out‚106948.36767578125 disk.sda3.in‚0' proc.memory.availableMB‚ 600.5390625 network.recv‚ 1.751081e+06 disk.sda.out‚ 186.1640625 disk.dm-0.in‚ 12650.46875# memory_percent‚96.3346004486084 cpu‚25.398335348082465 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 186.1640625" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17% disk./.usageGB‚1180.2453422546387 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7578125gÒÖ“æ‚(:ˆ €íëÔ¾þË gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚162.471' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ã.yç‚(:î €íëÔã†öÏ) disk.nvme0c0n1.out‚107269.93798828125 disk.nvme0n1.in‚ 12650.46875 disk.sda.out‚ 186.765625 disk.dm-0.in‚ 12650.46875$ memory_percent‚96.33462429046631 disk.drbd1260.out‚ 0.7578125' disk.nvme0n1.out‚107269.93798828125 disk.sda3.in‚0 disk.sda3.out‚ 186.765625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 network.sent‚ 9.695412e+06% disk./.usageGB‚1180.2443237304688! disk.nvme0c0n1.in‚ 12650.46875$ disk.dm-0.out‚107269.93798828125( proc.memory.availableMB‚ 600.53515625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 186.765625" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 cpu‚25.397548468423494 network.recv‚ 1.755579e+06) disk./.usagePercent‚79.20718685649946 disk.sda2.in‚0Råx “è‚(:ˆ íëÔ³ÓâË gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚115.281' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0562 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥›2€é‚(:õ íëÔ¥´ÄÌ network.sent‚ 9.701609e+06 network.recv‚ 1.760148e+06% disk./.usageGB‚1180.2443237304688 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 12650.46875 disk.dm-0.in‚ 12650.46875$ memory_percent‚96.33462429046631) disk./.usagePercent‚79.20718685649946 disk.drbd1260.out‚ 0.76171875) disk.nvme0c0n1.out‚107513.12158203125 disk.nvme0n1.in‚ 12650.46875' disk.nvme0n1.out‚107513.12158203125 disk.sda3.out‚ 187.33984375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 187.33984375 disk.dm-1.out‚ 187.33984375( proc.memory.availableMB‚ 600.53515625" proc.memory.rssMB‚ 3053.1171875 cpu‚25.396801419553277 disk.sda1.out‚0$ disk.dm-0.out‚107513.12158203125* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17' y2ê- _wandbruntime‚5366  _runtime‚5366Q±„ÆëÀ trainer/global_step‚3849 $ _timestamp‚1.788540570915779e+09 _runtime‚5366.226922781 _step‚313 & lr-AdamW/pg1‚0.0014523040946679401 & lr-AdamW/pg2‚0.0014523040946679401Cc5‘ËìÅ $ _timestamp‚1.788540570915779e+09 _runtime‚5366.226922781 _step‚313 & lr-AdamW/pg1‚0.0014523040946679401 & lr-AdamW/pg2‚0.0014523040946679401 trainer/global_step‚3849¹ÜËc2í- _wandbruntime‚5366  _runtime‚5366´QþUò'îìO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324512648649216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328923628261888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23431818 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97518600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611503032320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324512648649216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328924556236288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362636047201792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318164612201320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4395412686324 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324512630646272 _step‚314 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4410997615616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4411907587072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48264516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313770124529360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80551756 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178070356 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4611265861632 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97544944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7885405709175537e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328923628261888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318165537215684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4411907587072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚5366.228913905 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4410997615616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313770107490000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178096702 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71696334 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344_Ça›Û$ïñO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48264516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313770124529360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80551756 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178070356 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4611265861632 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97544944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7885405709175537e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328923628261888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚318165537215684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4411907587072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 _runtime‚5366.228913905 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4410997615616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313770107490000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178096702 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71696334 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324512648649216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328923628261888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23431818 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97518600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611503032320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324512648649216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328924556236288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362636047201792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318164612201320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4395412686324 F@DeviceStatsÏà&ÆMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324512630646272 _step‚314 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4410997615616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4411907587072 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144ºóèì—2ð- _wandbruntime‚5367  _runtime‚5367€Ï³ü&ñöM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97543431 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _step‚315 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313849125084544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324594267735040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚3849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4412119075840 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178141719 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4396529959810 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329007296795648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4612421455872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97569801 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97543431 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80571918 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178141719 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362727167389696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362727428487168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71714128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318244747069142 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4413029060608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23438177 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885405726638315e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324594249732096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178115347 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358114745933824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856Gñ‘'òûM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178141719 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4396529959810 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329007296795648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4612421455872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97569801 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97543431 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80571918 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178141719 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362727167389696 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362727428487168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71714128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318244747069142 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4413029060608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23438177 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885405726638315e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324594249732096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178115347 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358114745933824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97543431 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 _step‚315 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313849125084544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324594267735040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 trainer/global_step‚3849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4412119075840 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81559756 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72588771 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23721611 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4467085040640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322171369183628 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885406305954487e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180323258 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322172294197992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98763502 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367205702394368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180349604 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23721005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81559756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48867766 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333060669978112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328593602940416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328593584937472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367205942533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98789846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328593602940416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328593584937472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4467995012096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4451289941432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4450381966428 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333060669978112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362535829391872 _step‚318 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81559758 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0–¹Ë÷'ƒñO _step‚318 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81559758 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 trainer/global_step‚3899 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317720987217200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4669896942592 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81559756 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72588771 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23721611 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4467085040640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚322171369183628 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240139264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885406305954487e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180323258 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚322172294197992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98763502 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367205702394368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180349604 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23721005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81559756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48867766 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333060669978112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328593602940416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328593584937472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367205942533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98789846 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328593602940416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328593584937472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4467995012096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216199168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4451289941432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4450381966428 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333060669978112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362535829391872¾+Bp‘2„- _wandbruntime‚5427  _runtime‚5427š©“û&…õM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367297352840192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4671297743872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23726743 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81579916 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362626055096320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322252429066022 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333143410524160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72606574 trainer/global_step‚3899 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180394621 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328675222026240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98788333 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98814703 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4451499239706 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48879833 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328675204023296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72607180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚319 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328675222026240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180368249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317800021851104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48879831 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367297110617088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333144338511872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81579918 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180394621 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4468206500864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚5427.149126412 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328675204023296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4671079460864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317800004811744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4469116485632 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4451499239706 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48879833 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328675204023296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72607180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚319 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328675222026240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180368249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317800021851104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48879831 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367297110617088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚333144338511872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81579918 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180394621 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4468206500864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 _runtime‚5427.149126412 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328675204023296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218283008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4671079460864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317800004811744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4469116485632 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72606574 trainer/global_step‚3899 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180394621 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328675222026240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98788333 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¿_5éÛ2‡- _wandbruntime‚5427  _runtime‚5427ôÏ FˆÀ /loss/FRt2/detection_loss‚2.3617856531943144 epoch‚3 trainer/global_step‚3899 0loss/SRqqt1/detection_loss‚3.017218904835837 0loss/SRqqt3/detection_loss‚3.716245581706365 % _timestamp‚1.7885406318399808e+09 /loss/FRt1/assignment_loss‚3.727898825859201 /loss/FBt2/detection_loss‚1.1863743394124704 _runtime‚5427.150936635 _step‚320 1loss/SRqqt2/detection_loss‚1.8855132095275386 %loss/total_loss‚45.63523839471063 /loss/FRt3/assignment_loss‚4.751858802402721 0loss/FRt4/assignment_loss‚2.0139406295493245 1loss/FBt2/assignment_loss‚0.25322191946186084 loss/FBt4/assignment_loss‚0 /loss/FRt2/assignment_loss‚2.385097967950921 1loss/SRqqt1/assignment_loss‚2.308691999741963 .loss/FRt3/detection_loss‚2.484772779026437 0loss/SRqqt4/detection_loss‚1.050198008985085 /loss/FBt1/detection_loss‚2.2929944495360055 /loss/FBt3/detection_loss‚1.6715762151094775 /loss/FRt1/detection_loss‚3.0472126243443323 /loss/FBt4/detection_loss‚0.6980319023132324 1loss/FBt1/assignment_loss‚0.38960803380096976 /loss/FRt4/detection_loss‚1.7170549327274784 2loss/SRqqt2/assignment_loss‚1.2690345338515696 0loss/SRqqt3/assignment_loss‚1.72533509756128 2loss/SRqqt4/assignment_loss‚1.0717116629831114 0loss/FBt3/assignment_loss‚0.6098603208291328“..K‰Å /loss/FRt1/assignment_loss‚3.727898825859201 /loss/FBt2/detection_loss‚1.1863743394124704 _runtime‚5427.150936635 _step‚320 1loss/SRqqt2/detection_loss‚1.8855132095275386 %loss/total_loss‚45.63523839471063 /loss/FRt3/assignment_loss‚4.751858802402721 0loss/FRt4/assignment_loss‚2.0139406295493245 1loss/FBt2/assignment_loss‚0.25322191946186084 loss/FBt4/assignment_loss‚0 /loss/FRt2/assignment_loss‚2.385097967950921 1loss/SRqqt1/assignment_loss‚2.308691999741963 .loss/FRt3/detection_loss‚2.484772779026437 0loss/SRqqt4/detection_loss‚1.050198008985085 /loss/FBt1/detection_loss‚2.2929944495360055 /loss/FBt3/detection_loss‚1.6715762151094775 /loss/FRt1/detection_loss‚3.0472126243443323 /loss/FBt4/detection_loss‚0.6980319023132324 1loss/FBt1/assignment_loss‚0.38960803380096976 /loss/FRt4/detection_loss‚1.7170549327274784 2loss/SRqqt2/assignment_loss‚1.2690345338515696 0loss/SRqqt3/assignment_loss‚1.72533509756128 2loss/SRqqt4/assignment_loss‚1.0717116629831114 0loss/FBt3/assignment_loss‚0.6098603208291328 /loss/FRt2/detection_loss‚2.3617856531943144 epoch‚3 trainer/global_step‚3899 0loss/SRqqt1/detection_loss‚3.017218904835837 0loss/SRqqt3/detection_loss‚3.716245581706365 % _timestamp‚1.7885406318399808e+09À¨ øKžŠ‚(:“ ÚíëÔ´ªµÌ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.355000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0⇞ƒ‹‚(:ø ÚíëÔ„€õÐ" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17# disk.dm-0.out‚109140.9599609375 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 189.8984375( disk.nvme0c0n1.out‚109140.9599609375( proc.memory.availableMB‚ 600.18359375) disk./.usagePercent‚79.21627589198759 disk.sda.out‚ 189.8984375 disk.dm-0.in‚12827.69921875 disk.dm-1.in‚0 disk.sda2.in‚0$ disk.nvme0c0n1.in‚12827.69921875 cpu‚25.393106651071133 disk.sda1.out‚0 network.sent‚ 9.844917e+06 network.recv‚ 1.782094e+06" disk.nvme0n1.in‚12827.69921875& disk.nvme0n1.out‚109140.9599609375 disk.sda3.in‚0 disk.sda3.out‚ 189.8984375 disk.drbd1260.in‚0% disk./.usageGB‚1180.3797569274902 disk.sda2.out‚0 disk.drbd1260.out‚ 0.76953125$ memory_percent‚96.33677005767822mžŒ‚(:“ éíëÔˆ¯ÑÌ gpu.0.gpu‚16.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚155.141' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.028200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÛa}‚(:ò éíëÔñ¥¤Ò disk.sda.out‚ 190.171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 600.18359375 network.sent‚ 9.939637e+06% disk./.usageGB‚1180.3562507629395$ disk.nvme0c0n1.in‚12968.30078125& disk.nvme0n1.out‚109611.1396484375 disk.sda3.out‚ 190.171875 disk.dm-0.in‚12968.30078125" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 network.recv‚ 1.787779e+06( disk./.usagePercent‚79.2146983735616 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# disk.dm-0.out‚109611.1396484375 disk.dm-1.out‚ 190.171875 cpu‚25.39243163726905( disk.nvme0c0n1.out‚109611.1396484375" disk.nvme0n1.in‚12968.30078125$ memory_percent‚96.33677005767822 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7734375 disk.sda.in‚0LjhžŽ‚(:“ øíëÔÜ’“Ì gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚128.818' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.763600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m³þŠ}‚(:ò øíëÔý×âÏ$ memory_percent‚96.33526802062988" proc.memory.rssMB‚ 3053.1171875 cpu‚25.391657599342768 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.792134e+06$ disk.nvme0c0n1.in‚12968.30078125( disk.nvme0c0n1.out‚109779.6396484375& disk.nvme0n1.out‚109779.6396484375 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 9.945445e+06( disk./.usagePercent‚79.2146945334486' proc.memory.availableMB‚ 600.4296875* proc.memory.percent‚18.634748458862305" disk.nvme0n1.in‚12968.30078125 disk.sda3.out‚ 190.546875 disk.sda.out‚ 190.546875 disk.dm-0.in‚12968.30078125 disk.dm-1.in‚0 disk.dm-1.out‚ 190.546875 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚1180.3561935424805 disk.sda.in‚0# disk.dm-0.out‚109779.6396484375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7734375+ þÓ‚(:ˆ ‡îëÔüÓàË gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.873' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7746 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–uYƒ‘‚(:ø ‡îëÔ·ˆ²Ï( proc.memory.availableMB‚ 600.18359375% disk./.usageGB‚1180.3797416687012 disk.dm-0.in‚12968.30078125# disk.dm-0.out‚110125.4501953125 disk.dm-1.in‚0 disk.dm-1.out‚ 190.8984375 disk.sda2.out‚0( disk.nvme0c0n1.out‚110125.4501953125" proc.memory.rssMB‚ 3053.1171875 network.sent‚ 9.951656e+06 network.recv‚ 1.796788e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.out‚ 0.77734375$ disk.nvme0c0n1.in‚12968.30078125 disk.sda3.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚18.634748458862305 cpu‚25.390959789098158 proc.cpu.threads‚17) disk./.usagePercent‚79.21627486795745 disk.sda.out‚ 190.8984375 disk.sda1.in‚0" disk.nvme0n1.in‚12968.30078125& disk.nvme0n1.out‚110125.4501953125 disk.sda3.out‚ 190.8984375$ memory_percent‚96.33677005767822Ÿ‡äƒ2’- _wandbruntime‚5485  _runtime‚5485 ¶'õÇ“Á % _timestamp‚1.7885406906858082e+09 _runtime‚5485.996966228 _step‚321 & lr-AdamW/pg1‚0.0014352502828266086 & lr-AdamW/pg2‚0.0014352502828266086 trainer/global_step‚3949ˆ”€¤Ì”Æ _runtime‚5485.996966228 _step‚321 & lr-AdamW/pg1‚0.0014352502828266086 & lr-AdamW/pg2‚0.0014352502828266086 trainer/global_step‚3949 % _timestamp‚1.7885406906858082e+09ÁdFŽ™2•- _wandbruntime‚5485  _runtime‚5485KŒv«ö'–ðO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49471018 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24012363 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73483381 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100008404 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337197711694336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4524082437120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚3949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82567758 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326178126165936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337198639668736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4524082437120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367047090674176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337198639668736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332674557231616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367047114614272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885406906875694e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100034748 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4506259221536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371775834190848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321671866944400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4728719576576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4728499183104 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332674539228672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73482775 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24011759 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337197711694336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332674539228672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73482775 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24011759 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337197711694336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49471018 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24012363 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73483381 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100008404 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337197711694336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4524082437120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 trainer/global_step‚3949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82567758 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326178126165936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337198639668736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4524082437120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367047090674176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337198639668736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332674557231616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367047114614272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 % _timestamp‚1.7885406906875694e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100034748 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4506259221536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371775834190848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321671866944400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4728719576576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_*ÉóÛ×pool.peak‚3725 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4728499183104 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326259186048330 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4507376494814 trainer/global_step‚3949 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332756158314496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4729665737216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4508284470026 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182621151 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367137340318720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182647523 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82587918 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4729894505984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49483081 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332756176317440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24017508 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321750901578304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4525203910656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82587916 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100059605 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337281380228096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4524293925888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332756176317440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182647523 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100059605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82587916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371867234824704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚5487.323216484 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337281380228096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100033235 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4524293925888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚323 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332756158314496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337280452240384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24018119 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326258261033758 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100033235 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182621151 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82587918 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73500589 % _timestamp‚1.7885406920111969e+09½'hé'šÿM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182647523 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100059605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82587916 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371867234824704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚5487.323216484 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚337281380228096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100033235 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4524293925888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚323 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332756158314496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337280452240384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24018119 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326258261033758 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100033235 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182621151 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82587918 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73500589 % _timestamp‚1.7885406920111969e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73501202 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367137316378624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326259186048330 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4507376494814 trainer/global_step‚3949 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332756158314496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4729665737216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4508284470026 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182621151 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367137340318720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182647523 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82587918 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4729894505984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49483081 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332756176317440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24017508 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321750901578304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4525203910656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82587916 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100059605 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337281380228096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4524293925888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332756176317440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ã2Ò›‰2›- _wandbruntime‚5487  _runtime‚5487~AÅIœà 1loss/SRqqt1/assignment_loss‚2.133215332951616 2loss/SRqqt3/assignment_loss‚2.1855102902964543 0loss/FBt2/assignment_loss‚0.1804443150493505 loss/FBt4/assignment_loss‚0 1loss/SRqqt4/detection_loss‚1.0206428574136466 .loss/FBt1/detection_loss‚2.424034363236921 /loss/FBt2/detection_loss‚1.5073801089002161 0loss/SRqqt3/detection_loss‚2.992105687919416 _step‚324 2loss/SRqqt2/assignment_loss‚1.4424811750650406 0loss/FBt3/assignment_loss‚0.3884772766647955 1loss/SRqqt2/detection_loss‚1.9250857532024384 &loss/total_loss‚43.610352978314666 $ _timestamp‚1.788540692014554e+09 _runtime‚5487.325482586 trainer/global_step‚3949 0loss/FRt1/assignment_loss‚3.2290269783564978 2loss/SRqqt4/assignment_loss‚1.2577164962306648 /loss/FBt4/detection_loss‚0.7713112831115723 epoch‚3 /loss/FRt1/detection_loss‚2.5603994735649653 .loss/FRt3/detection_loss‚2.687079387704026 1loss/SRqqt1/detection_loss‚2.2656136610928703 /loss/FBt3/detection_loss‚1.9944936103290982 0loss/FRt3/assignment_loss‚3.5624891110828947 0loss/FBt1/assignment_loss‚0.3245331941411834 /loss/FRt2/detection_loss‚2.5110043417114545 0loss/FRt4/assignment_loss‚1.6104719202909896 /loss/FRt4/detection_loss‚1.6996518491364236 /loss/FRt2/assignment_loss‚2.937184510862126s9˃NÈ 0loss/FBt2/assignment_loss‚0.1804443150493505 loss/FBt4/assignment_loss‚0 1loss/SRqqt4/detection_loss‚1.0206428574136466 .loss/FBt1/detection_loss‚2.424034363236921 /loss/FBt2/detection_loss‚1.5073801089002161 0loss/SRqqt3/detection_loss‚2.992105687919416 _step‚324 2loss/SRqqt2/assignment_loss‚1.4424811750650406 0loss/FBt3/assignment_loss‚0.3884772766647955 1loss/SRqqt2/detection_loss‚1.9250857532024384 &loss/total_loss‚43.610352978314666 $ _timestamp‚1.788540692014554e+09 _runtime‚5487.325482586 trainer/global_step‚3949 0loss/FRt1/assignment_loss‚3.2290269783564978 2loss/SRqqt4/assignment_loss‚1.2577164962306648 /loss/FBt4/detection_loss‚0.7713112831115723 epoch‚3 /loss/FRt1/detection_loss‚2.5603994735649653 .loss/FRt3/detection_loss‚2.687079387704026 1loss/SRqqt1/detection_loss‚2.2656136610928703 /loss/FBt3/detection_loss‚1.9944936103290982 0loss/FRt3/assignment_loss‚3.5624891110828947 0loss/FBt1/assignment_loss‚0.3245331941411834 /loss/FRt2/detection_loss‚2.5110043417114545 0loss/FRt4/assignment_loss‚1.6104719202909896 /loss/FRt4/detection_loss‚1.6996518491364236 /loss/FRt2/assignment_loss‚2.937184510862126 1loss/SRqqt1/assignment_loss‚2.133215332951616 2loss/SRqqt3/assignment_loss‚2.1855102902964543Ä›¶ 4“ž‚(:ˆ –îëÔ¹è“Ì gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.844' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.1688 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ@—„Ÿ‚(:ù –îëÔž¹ÛÎ network.sent‚ 9.956749e+06 network.recv‚ 1.800212e+06& disk.nvme0n1.out‚110427.1064453125 disk.sda.in‚0 disk.sda.out‚ 191.78515625 disk.sda1.out‚0$ memory_percent‚96.33855819702148) disk./.usagePercent‚79.21627486795745 disk.sda2.in‚0$ disk.nvme0c0n1.in‚12968.30078125" disk.nvme0n1.in‚12968.30078125 disk.sda3.in‚0 disk.dm-0.in‚12968.30078125# disk.dm-0.out‚110427.1064453125 disk.dm-1.out‚ 191.78515625 disk.sda2.out‚0 disk.sda3.out‚ 191.78515625 disk.sda1.in‚0& proc.memory.availableMB‚ 599.890625" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17% disk./.usageGB‚1180.3797416687012 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.77734375( disk.nvme0c0n1.out‚110427.1064453125 disk.dm-1.in‚0 cpu‚25.390223233913634Wß»“ ‚(:ˆ ¥îëÔÚ¡˜Ê gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³c?‚¡‚(:÷ ¥îëÔ•®èË disk.sda1.in‚0 disk.dm-1.out‚ 192.62109375 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚13074.72265625$ memory_percent‚96.33550643920898 network.sent‚ 1.0063542e+07% disk./.usageGB‚1180.3562545776367 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-0.in‚13074.72265625# disk.dm-0.out‚110562.3642578125& proc.memory.availableMB‚ 600.390625" proc.memory.rssMB‚ 3053.1171875 disk.sda3.out‚ 192.62109375 disk.sda.in‚0 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚13074.72265625( disk.nvme0c0n1.out‚110562.3642578125* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17 disk.sda.out‚ 192.62109375 disk.dm-1.in‚0 disk.drbd1260.out‚0.78125& disk.nvme0n1.out‚110562.3642578125 cpu‚25.389466762776056 network.recv‚ 1.804926e+06) disk./.usagePercent‚79.21469862956914Jì!º’¢‚(:‡ ´îëÔ¹¶ÒÒ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚165.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 d~£‚(:ó ´îëÔ£¥ìØ disk.sda1.in‚0 disk.dm-1.out‚ 192.953125" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 network.sent‚ 1.0069715e+07 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚13111.62109375& disk.nvme0n1.out‚110835.8291015625 disk.sda3.out‚ 192.953125 disk.sda.out‚ 192.953125 disk.dm-0.in‚13111.62109375) disk./.usagePercent‚79.21469862956914 disk.drbd1260.in‚0" disk.nvme0n1.in‚13111.62109375 disk.sda.in‚0 disk.sda1.out‚0# disk.dm-0.out‚110835.8291015625 disk.dm-1.in‚0( proc.memory.availableMB‚ 600.14453125( disk.nvme0c0n1.out‚110835.8291015625$ memory_percent‚96.33700847625732 cpu‚25.388788729677497 proc.cpu.threads‚17 network.recv‚ 1.809422e+06% disk./.usageGB‚1180.3562545776367 disk.drbd1260.out‚0.78125 disk.sda3.in‚0›_!铤‚(:ˆ ÃîëÔüúíÊ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØïÉÝ€¥‚(:õ ÃîëÔžà€Î$ disk.nvme0c0n1.in‚13111.62109375 disk.sda3.out‚ 193.96875 disk.sda.in‚0$ disk.dm-0.out‚111216.35009765625 disk.drbd1260.in‚0 disk.sda.out‚ 193.96875 cpu‚25.38806990002225 proc.cpu.threads‚17 network.sent‚ 1.0075925e+07 network.recv‚ 1.814334e+06) disk./.usagePercent‚79.21463949182873 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚13111.62109375' disk.nvme0n1.out‚111216.35009765625$ memory_percent‚96.33691310882568 disk.dm-0.in‚13111.62109375) disk.nvme0c0n1.out‚111216.35009765625 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 600.16015625" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305% disk./.usageGB‚1180.3553733825684 disk.dm-1.in‚0 disk.dm-1.out‚ 193.96875 disk.drbd1260.out‚ 0.78515625R2¦- _wandbruntime‚5546  _runtime‚5546‚PÀ?ǧÁ trainer/global_step‚3999 % _timestamp‚1.7885407514611106e+09 _runtime‚5546.772170889 _step‚325 & lr-AdamW/pg1‚0.0014265372664843338 & lr-AdamW/pg2‚0.0014265372664843338B ÁÌ¨Æ & lr-AdamW/pg2‚0.0014265372664843338 trainer/global_step‚3999 % _timestamp‚1.7885407514611106e+09 _runtime‚5546.772170889 _step‚325 & lr-AdamW/pg1‚0.0014265372664843338Åu¯ß¡2©- _wandbruntime‚5546  _runtime‚5546 ±BW ªðO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚341334753410560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24302094 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376345931096064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371558375896576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184829062 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83575758 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336755493519872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336755493519872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184855408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341334753410560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325622746671600 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101279650 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336755511522816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4787302477312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101279650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4562136476644 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341335681384960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4563044451648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4579259890688 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50074266 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4579259890688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74376360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341334753410560 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚325622746671600 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101279650 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336755511522816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4787302477312 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101279650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252722176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4562136476644 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341335681384960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4563044451648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4579259890688 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50074266 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4579259890688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74376360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336755493519872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336755493519872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184855408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184900425 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4580381350912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83595916 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24308568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184874053 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336837112605696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341417493956608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4581291335680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚5547.897634057 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83595916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74394258 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376437320098304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4788694557184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4788461594112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330265018016066 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4563253749922 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4564161725134 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚3999 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336837130608640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101278137 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376437063195136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341418421944320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184874053 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚325701764266144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚371648625541120 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184900425 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325701781305504 % _timestamp‚1.7885407525861638e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50086331 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184900425 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4580381350912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83595916 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 _step‚327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24308568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184874053 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336837112605696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341417493956608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/invôLôactive_split.large_pool.peak‚44 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4581291335680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 _runtime‚5547.897634057 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83595916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74394258 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376437320098304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4788694557184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4788461594112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330265018016066 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4563253749922 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4564161725134 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 trainer/global_step‚3999 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336837130608640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101278137 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚376437063195136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570ÇMJ&2¯- _wandbruntime‚5547  _runtime‚5547œÓçÑD°¾ trainer/global_step‚3999 /loss/FRt1/detection_loss‚2.4128965394094934 /loss/FRt1/assignment_loss‚3.937765307204668 _runtime‚5547.899649626 2loss/SRqqt2/assignment_loss‚1.3369443606585265 2loss/SRqqt4/assignment_loss‚1.1222576367920332 1loss/SRqqt2/detection_loss‚2.4173934757709503 1loss/SRqqt4/detection_loss‚0.8421419085866778 /loss/FRt2/assignment_loss‚2.607753779206957 2loss/SRqqt3/assignment_loss‚1.6094120383262633 0loss/FBt2/assignment_loss‚0.2922857788115999 1loss/SRqqt3/detection_loss‚2.6260001252094907 /loss/FBt2/detection_loss‚0.9175125735908233 epoch‚3 ,loss/FRt3/assignment_loss‚4.004441122214 0loss/FRt4/assignment_loss‚1.7379947658628225 .loss/FRt4/detection_loss‚1.814637009636499 0loss/SRqqt1/detection_loss‚2.532176721841097 /loss/FBt1/detection_loss‚1.9094046003052167 /loss/FBt4/detection_loss‚0.5249524116516113 1loss/SRqqt1/assignment_loss‚1.854946224950254 .loss/FRt3/detection_loss‚2.585171375795846 % _timestamp‚1.7885407525888498e+09 .loss/FRt2/detection_loss‚2.436201992843832 1loss/FBt1/assignment_loss‚0.23207080691801302 0loss/FBt3/assignment_loss‚0.6500031810533723 loss/FBt4/assignment_loss‚0 %loss/total_loss‚41.99480696922461 /loss/FBt3/detection_loss‚1.5904432325845672 _step‚328…DH#I±à /loss/FBt1/detection_loss‚1.9094046003052167 /loss/FBt4/detection_loss‚0.5249524116516113 1loss/SRqqt1/assignment_loss‚1.854946224950254 .loss/FRt3/detection_loss‚2.585171375795846 % _timestamp‚1.7885407525888498e+09 .loss/FRt2/detection_loss‚2.436201992843832 1loss/FBt1/assignment_loss‚0.23207080691801302 0loss/FBt3/assignment_loss‚0.6500031810533723 loss/FBt4/assignment_loss‚0 %loss/total_loss‚41.99480696922461 /loss/FBt3/detection_loss‚1.5904432325845672 _step‚328 trainer/global_step‚3999 /loss/FRt1/detection_loss‚2.4128965394094934 /loss/FRt1/assignment_loss‚3.937765307204668 _runtime‚5547.899649626 2loss/SRqqt2/assignment_loss‚1.3369443606585265 2loss/SRqqt4/assignment_loss‚1.1222576367920332 1loss/SRqqt2/detection_loss‚2.4173934757709503 1loss/SRqqt4/detection_loss‚0.8421419085866778 /loss/FRt2/assignment_loss‚2.607753779206957 2loss/SRqqt3/assignment_loss‚1.6094120383262633 0loss/FBt2/assignment_loss‚0.2922857788115999 1loss/SRqqt3/detection_loss‚2.6260001252094907 /loss/FBt2/detection_loss‚0.9175125735908233 epoch‚3 ,loss/FRt3/assignment_loss‚4.004441122214 0loss/FRt4/assignment_loss‚1.7379947658628225 .loss/FRt4/detection_loss‚1.814637009636499 0loss/SRqqt1/detection_loss‚2.532176721841097È×ax8’²‚(:‡ ÒîëÔ’¾¨Ê gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.315 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"R?M…³‚(:ú ÒîëÔ÷›“Í disk.sda2.in‚0" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17) disk./.usagePercent‚79.21470784584037' disk.nvme0n1.out‚111537.97119140625 disk.sda3.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 599.84765625 cpu‚25.38744013106849 network.sent‚ 1.0081001e+07 network.recv‚ 1.818091e+06$ disk.nvme0c0n1.in‚13111.62109375 disk.sda.in‚0 disk.dm-0.in‚13111.62109375$ disk.dm-0.out‚111537.97119140625 disk.drbd1260.out‚ 0.78515625# memory_percent‚96.3388204574585) disk.nvme0c0n1.out‚111537.97119140625 disk.sda.out‚ 194.8828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 194.8828125 disk.sda2.out‚0% disk./.usageGB‚1180.3563919067383" disk.nvme0n1.in‚13111.62109375 disk.sda3.out‚ 194.8828125#ø ý“´‚(:ˆ áîëÔ…ØË gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.7166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y‚A¾‚µ‚(:÷ áîëÔÆ¢Ñ) disk./.usagePercent‚79.21627998810814 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚13111.62109375 disk.sda3.out‚ 195.48828125 disk.sda.in‚0 disk.dm-1.in‚0# memory_percent‚96.3388442993164 network.sent‚ 1.0190576e+07 disk.drbd1260.out‚ 0.7890625( disk.nvme0c0n1.out‚111883.7900390625 disk.sda.out‚ 195.48828125 disk.dm-0.in‚13111.62109375# disk.dm-0.out‚111883.7900390625% proc.memory.availableMB‚ 599.84375* proc.memory.percent‚18.634748458862305% disk./.usageGB‚1180.3798179626465" disk.nvme0n1.in‚13111.62109375 disk.sda3.in‚0 disk.dm-1.out‚ 195.48828125 cpu‚25.386724323584556 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚111883.7900390625 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3053.1171875 network.recv‚ 1.823623e+06ö§›ž¶‚(:“ ðîëÔ´ŸÌ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.699' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.339800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž_þgx·‚(:í ðîëÔž§ºÏ( disk./.usagePercent‚79.2162774280328$ disk.nvme0c0n1.in‚13254.37109375 disk.dm-0.in‚13254.37109375 disk.dm-1.in‚0 disk.dm-1.out‚196.1875$ memory_percent‚96.33736610412598 cpu‚25.385975338539602 network.sent‚ 1.0194561e+07 network.recv‚ 1.828274e+06 disk.sda2.in‚0 disk.drbd1260.out‚ 0.7890625 disk.sda.in‚0" proc.memory.rssMB‚ 3053.1171875 proc.cpu.threads‚17% disk./.usageGB‚1180.3797798156738" disk.nvme0n1.in‚13254.37109375 disk.sda3.in‚0 disk.sda.out‚196.1875 disk.sda1.in‚0# disk.dm-0.out‚112094.8759765625' proc.memory.availableMB‚ 600.0859375* proc.memory.percent‚18.634748458862305 disk.sda3.out‚196.1875 disk.sda2.out‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚112094.8759765625& disk.nvme0n1.out‚112094.8759765625 disk.sda1.out‚0"Û’¸‚(:‡ ÿîëÔ‰Ð˜Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì‹æZŠ¹‚(:ÿ ÿîëÔñüøÏ" proc.memory.rssMB‚ 3053.1171875 cpu‚25.385299405146355 network.sent‚ 1.0200862e+07% disk./.usageGB‚1180.3798179626465 disk.dm-0.in‚13254.37109375$ disk.dm-0.out‚112248.59619140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.79296875$ disk.nvme0c0n1.in‚13254.37109375' disk.nvme0n1.out‚112248.59619140625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 197.01953125 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚96.33734226226807* proc.memory.percent‚18.634748458862305 network.recv‚ 1.832737e+06 disk.sda3.out‚ 197.01953125 disk.sda1.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 600.08984375 proc.cpu.threads‚17) disk./.usagePercent‚79.21627998810814" disk.nvme0n1.in‚13254.37109375 disk.dm-1.out‚ 197.01953125 disk.sda2.out‚0) disk.nvme0c0n1.out‚112248.59619140625ÚÖR>2º- _wandbruntime‚5606  _runtime‚5606óç5Ç»Á _runtime‚5606.708550198 _step‚329 & lr-AdamW/pg1‚0.0014177025889895787 & lr-AdamW/pg2‚0.0014177025889895787 trainer/global_step‚4049 % _timestamp‚1.7885408113963182e+09‡û1äÌ¼Æ _step‚329 & lr-AdamW/pg1‚0.0014177025889895787 & lr-AdamW/pg2‚0.0014177025889895787 trainer/global_step‚4049 % _timestamp‚1.7885408113963182e+09 _runtime‚5606.708550198É!8$2½- _wandbruntime‚5606  _runtime‚5606„}’›õ'¾ïO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380915708426240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187108310 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376069661118976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚329573626398800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340836447811072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345471795126784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84583758 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75271117 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75270482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24592966 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4618921706756 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187081964 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102524552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102498208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24593599 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4635347315712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329573643438160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4636257287168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345471795126784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4846023367168 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚4049 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788540811401204e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84583756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50677516 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345472723101184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340836465814016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102498208 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340836465814016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4636257287168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4635347315712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380915447315456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84583758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334192565144916 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334191640130552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5606.713426421 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50677518 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84583756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376069685059072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400ðYº…%¿ôO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345471795126784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4846023367168 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 trainer/global_step‚4049 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788540811401204e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84583756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50677516 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345472723101184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340836465814016 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102498208 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340836465814016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4636257287168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4635347315712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380915447315456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84583758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334192565144916 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334191640130552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5606.713426421 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50677518 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84583756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376069685059072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380915708426240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187108310 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376069661118976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 _step‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚329573626398800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340836447811072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345471795126784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84583758 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75271117 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75270482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24592966 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4618921706756 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187081964 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102524552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102498208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24593599 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_poo–ü*Tul.peak‚ 353397248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4635347315712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329573643438160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4636257287168Ê2œéÙ2À- _wandbruntime‚5608  _runtime‚5608ø—'ÁúM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345555463660544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102523039 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334272700012946 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4620038980242 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102549409 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84603918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50689583 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334271774998374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885408128068519e+09 _runtime‚5608.118776798 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4637378760704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187126955 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345554535672832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340918084899840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4636468775936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187153327 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84603918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24598814 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381006850596352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75289018 trainer/global_step‚4049 _step‚331 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329652661032704 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340918066896896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340918084899840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102523039 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75288395 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50689581 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187153327 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615ï,5­'ÂÿM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345554535672832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340918084899840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4636468775936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187153327 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84603918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24598814 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381006850596352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75289018 trainer/global_step‚4049 _step‚331 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329652661032704 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340918066896896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚340918084899840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102523039 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75288395 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50689581 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187153327 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚345555463660544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102523039 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334272700012946 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4620038980242 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102549409 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84603918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50689583 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334271774998374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885408128068519e+09 _runtime‚5608.118776798 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4637378760704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187126955 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224574464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ËFëp“2Ã- _wandbruntime‚5608  _runtime‚5608p=FâVÄÐ 0loss/FBt3/assignment_loss‚0.5220483636211637 2loss/SRqqt2/assignment_loss‚1.6269593155011535 .loss/FRt3/detection_loss‚2.534430728809005 1loss/SRqqt1/detection_loss‚2.4427326560020446 1loss/SRqqt2/detection_loss‚2.3001814944048724 /loss/FBt1/detection_loss‚2.5302815186349967 /loss/FBt3/detection_loss‚1.9372717812657356 /loss/FRt2/assignment_loss‚2.963575871979318 /loss/FBt4/detection_loss‚6.4775402545928955 $ _timestamp‚1.788540812808708e+09 2loss/SRqqt1/assignment_loss‚2.2471866885821026 1loss/FBt2/assignment_loss‚0.21296488795703758 .loss/FRt4/detection_loss‚1.630905602578864 %loss/total_loss‚52.95631855781291 _step‚332 2loss/SRqqt3/assignment_loss‚1.7254478058644704 _runtime‚5608.119704667 .loss/FRt3/assignment_loss‚4.03922798055591 0loss/FRt4/assignment_loss‚1.7675316297646724 2loss/SRqqt4/assignment_loss‚1.3729301300372754 0loss/FBt1/assignment_loss‚0.2362057108065959 1loss/SRqqt4/detection_loss‚0.8489586874493398 /loss/FBt2/detection_loss‚0.9802860943921681 /loss/FRt1/detection_loss‚2.4910214767363428 .loss/FRt2/detection_loss‚2.323240188927185 epoch‚3 trainer/global_step‚4049 /loss/FRt1/assignment_loss‚3.120454079396016 /loss/FBt4/assignment_loss‚2.630897283554077 0loss/SRqqt3/detection_loss‚3.994038326399667î¿4R[ÅÕ .loss/FRt4/detection_loss‚1.630905602578864 %loss/total_loss‚52.95631855781291 _step‚332 2loss/SRqqt3/assignment_loss‚1.7254478058644704 _runtime‚5608.119704667 .loss/FRt3/assignment_loss‚4.03922798055591 0loss/FRt4/assignment_loss‚1.7675316297646724 2loss/SRqqt4/assignment_loss‚1.3729301300372754 0loss/FBt1/assignment_loss‚0.2362057108065959 1loss/SRqqt4/detection_loss‚0.8489586874493398 /loss/FBt2/detection_loss‚0.9802860943921681 /loss/FRt1/detection_loss‚2.4910214767363428 .loss/FRt2/detection_loss‚2.323240188927185 epoch‚3 trainer/global_step‚4049 /loss/FRt1/assignment_loss‚3.120454079396016 /loss/FBt4/assignment_loss‚2.630897283554077 0loss/SRqqt3/detection_loss‚3.994038326399667 0loss/FBt3/assignment_loss‚0.5220483636211637 2loss/SRqqt2/assignment_loss‚1.6269593155011535 .loss/FRt3/detection_loss‚2.534430728809005 1loss/SRqqt1/detection_loss‚2.4427326560020446 1loss/SRqqt2/detection_loss‚2.3001814944048724 /loss/FBt1/detection_loss‚2.5302815186349967 /loss/FBt3/detection_loss‚1.9372717812657356 /loss/FRt2/assignment_loss‚2.963575871979318 /loss/FBt4/detection_loss‚6.4775402545928955 $ _timestamp‚1.788540812808708e+09 2loss/SRqqt1/assignment_loss‚2.2471866885821026 1loss/FBt2/assignment_loss‚0.21296488795703758ÌÌZ>“Æ‚(:ˆ ŽïëÔæ¯ÔÎ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\>‹vˆÇ‚(:ý ŽïëÔɵ¬Ð" disk.nvme0n1.in‚13254.37109375 disk.sda3.in‚0 disk.sda1.in‚0 network.sent‚ 1.0205945e+07 disk.dm-1.in‚0 disk.dm-1.out‚ 197.98828125 disk.sda2.out‚0' disk.nvme0n1.out‚112627.95556640625 disk.sda3.out‚ 197.98828125 disk.sda.out‚ 197.98828125 disk.sda1.out‚0 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚112627.95556640625$ memory_percent‚96.33762836456299( proc.memory.availableMB‚ 600.04296875 cpu‚25.3846276021774 proc.cpu.threads‚17% disk./.usageGB‚1180.3798370361328 disk.dm-0.in‚13254.37109375$ disk.dm-0.out‚112627.95556640625 disk.sda.in‚0" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 network.recv‚ 1.835928e+06) disk./.usagePercent‚79.21628126814582 disk.sda2.in‚0 disk.drbd1260.out‚ 0.79296875$ disk.nvme0c0n1.in‚13254.37109375§YDˆ“È‚(:ˆ ïëÔÙÓË gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\éÌôƒÉ‚(:ø ïëÔÉÅÎ% disk./.usageGB‚1180.3563499450684& disk.nvme0n1.out‚113103.5380859375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 599.8046875( disk./.usagePercent‚79.2147050297575( disk.nvme0c0n1.out‚113103.5380859375# disk.dm-0.out‚113103.5380859375 disk.dm-1.out‚ 198.31640625$ memory_percent‚96.33908271789551* proc.memory.percent‚18.634748458862305 cpu‚25.384024004649387 proc.cpu.threads‚17 network.recv‚ 1.841216e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚0.796875" disk.nvme0n1.in‚13254.37109375 disk.sda3.out‚ 198.31640625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.0313516e+07$ disk.nvme0c0n1.in‚13254.37109375 disk.sda.out‚ 198.31640625 disk.dm-0.in‚13254.37109375 disk.sda2.out‚0" proc.memory.rssMB‚ 3053.1171875B¬£Ê‚(:’ ¬ïëÔ·—½Í gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ ±dzË‚(:ï ¬ïëÔ¡ÄÐÒ network.recv‚ 1.845565e+06% disk./.usageGB‚1180.3562927246094 disk.sda1.out‚0' proc.memory.availableMB‚ 600.0546875" proc.memory.rssMB‚ 3053.1171875 cpu‚25.38332632005025 proc.cpu.threads‚17& disk.nvme0n1.out‚113256.8310546875 disk.sda1.in‚0 disk.dm-0.in‚ 13427.859375# disk.dm-0.out‚113256.8310546875 disk.sda2.out‚0 network.sent‚ 1.0319424e+07) disk./.usagePercent‚79.21470118964449 disk.sda3.out‚ 198.9296875 disk.sda.out‚ 198.9296875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚0.796875" disk.nvme0c0n1.in‚ 13427.859375 disk.nvme0n1.in‚ 13427.859375 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 198.9296875 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚113256.8310546875$ memory_percent‚96.33755683898926* proc.memory.percent‚18.634748458862305}ZuW“Ì‚(:ˆ »ïëÔààÆÊ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!‹ü}Í‚(:ò »ïëÔ·¼­Ì) disk.nvme0c0n1.out‚113732.48583984375' disk.nvme0n1.out‚113732.48583984375 disk.sda.in‚0 disk.sda.out‚ 199.390625 disk.dm-0.in‚ 13427.859375$ memory_percent‚96.34053707122803 cpu‚25.38268172872251 disk.sda3.out‚ 199.390625 disk.sda1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3053.1171875 network.sent‚ 1.0325624e+07 disk.nvme0n1.in‚ 13427.859375 disk.sda3.in‚0$ disk.dm-0.out‚113732.48583984375 disk.dm-1.in‚0 disk.dm-1.out‚ 199.390625( proc.memory.availableMB‚ 599.56640625 proc.cpu.threads‚17 network.recv‚ 1.850306e+06) disk./.usagePercent‚79.21627998810814" disk.nvme0c0n1.in‚ 13427.859375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.80078125* proc.memory.percent‚18.634748458862305% disk./.usageGB‚1180.37981796264654Ð{¡2Î- _wandbruntime‚5667  _runtime‚5667ÌðéÅÏ¿ & lr-AdamW/pg1‚0.0014087480914758916 & lr-AdamW/pg2‚0.0014087480914758916 trainer/global_step‚4099 # _timestamp‚1.78854087187794e+09 _runtime‚5667.188899824 _step‚333m;…ÊÐÄ & lr-AdamW/pg2‚0.0014087480914758916 trainer/global_step‚4099 # _timestamp‚1.78854087187794e+09 _runtime‚5667.188899824 _step‚333 & lr-AdamW/pg1‚0.0014087480914758916ͬ I‚2Ñ- _wandbruntime‚5667  _runtime‚56670Ô"õš ÒðO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4691434740736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4692344712192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 261110784 _step‚334 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85591756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51280766 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚385485736376832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85591758 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚344917420105216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885408718791313e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380580970281472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344917420105216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24883278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385485475266048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338198397112860 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333524523165360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349608836843008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189361212 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103743110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51280768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85591758 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103743110 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338199322127224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349609764817408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24883901 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4674798961864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4673890986860 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76164044 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189334866 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚344917420105216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 % _timestamp‚1.7885408718791313e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380580970281472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344917420105216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24883278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385485475266048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4099 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338198397112860 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333524523165360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349608836843008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189361212 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237170688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103743110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51280768 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85591758 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103743110 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚338199322127224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349609764817408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24883901 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4674798961864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4673890986860 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76164044 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189334866 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380671172045824 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51292831 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349692505376768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24889623 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24889004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344999039191040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4692556200960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189379857 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338278531980682 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85611916 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338279456995254 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189406229 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85611918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51292833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385577128492032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚333603540759904 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76181835 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚385576877880320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344999021188096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 % _timestamp‚1.7885408733552895e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103794311 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4693466185728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4693466185728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380671195985920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344999039191040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85611918 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85611916 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338279456995254 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189406229 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85611918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51292833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚385577128492032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚333603540759904 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76181835 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚385576877880320 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344999021188096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 % _timestamp‚1.7885408733552895e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103794311 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4693466185728 \JDeviceStatsMonitor.on_tr­0ü\±ain_batch_end/allocated_bytes.small_pool.allocated‚ 4693466185728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380671195985920 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344999039191040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85611918 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380671172045824Ïô#[û2×- _wandbruntime‚5668  _runtime‚5668Õ® ZØÔ .loss/FBt1/detection_loss‚2.366056776046753 0loss/FRt4/assignment_loss‚1.9804835796356202 1loss/SRqqt3/assignment_loss‚2.408732056617737 _runtime‚5668.669414792 0loss/FBt4/assignment_loss‚2.6904261112213135 _step‚336 0loss/FRt3/assignment_loss‚3.4553591935407546 /loss/FRt3/detection_loss‚2.4209501905593025 1loss/SRqqt1/detection_loss‚2.5821522005966733 1loss/SRqqt2/detection_loss‚1.9822594925113346 /loss/FBt2/detection_loss‚1.1584727126410452 /loss/FRt1/assignment_loss‚3.198843079805374 0loss/FRt2/assignment_loss‚3.1047989565592546 2loss/SRqqt2/assignment_loss‚1.7267299060932482 .loss/FRt4/detection_loss‚1.814740110312899 1loss/SRqqt3/detection_loss‚2.5921906016089697 /loss/FBt3/detection_loss‚1.7503343175711303 0loss/FBt3/assignment_loss‚0.6034731063705765 .loss/FRt2/detection_loss‚2.806308225943492 2loss/SRqqt4/assignment_loss‚1.1271949720026955 .loss/FRt1/detection_loss‚2.254082740172744 1loss/SRqqt4/detection_loss‚0.9919200184159591 .loss/FBt4/detection_loss‚6.338654279708862 %loss/total_loss‚51.84189833680588 trainer/global_step‚4099 1loss/FBt1/assignment_loss‚0.34072094571286776 epoch‚3 % _timestamp‚1.7885408733585703e+09 2loss/SRqqt1/assignment_loss‚1.8642264689717973 1loss/FBt2/assignment_loss‚0.28278829418547596*‹ê_ÙÙ 0loss/FBt4/assignment_loss‚2.6904261112213135 _step‚336 0loss/FRt3/assignment_loss‚3.4553591935407546 /loss/FRt3/detection_loss‚2.4209501905593025 1loss/SRqqt1/detection_loss‚2.5821522005966733 1loss/SRqqt2/detection_loss‚1.9822594925113346 /loss/FBt2/detection_loss‚1.1584727126410452 /loss/FRt1/assignment_loss‚3.198843079805374 0loss/FRt2/assignment_loss‚3.1047989565592546 2loss/SRqqt2/assignment_loss‚1.7267299060932482 .loss/FRt4/detection_loss‚1.814740110312899 1loss/SRqqt3/detection_loss‚2.5921906016089697 /loss/FBt3/detection_loss‚1.7503343175711303 0loss/FBt3/assignment_loss‚0.6034731063705765 .loss/FRt2/detection_loss‚2.806308225943492 2loss/SRqqt4/assignment_loss‚1.1271949720026955 .loss/FRt1/detection_loss‚2.254082740172744 1loss/SRqqt4/detection_loss‚0.9919200184159591 .loss/FBt4/detection_loss‚6.338654279708862 %loss/total_loss‚51.84189833680588 trainer/global_step‚4099 1loss/FBt1/assignment_loss‚0.34072094571286776 epoch‚3 % _timestamp‚1.7885408733585703e+09 2loss/SRqqt1/assignment_loss‚1.8642264689717973 1loss/FBt2/assignment_loss‚0.28278829418547596 .loss/FBt1/detection_loss‚2.366056776046753 0loss/FRt4/assignment_loss‚1.9804835796356202 1loss/SRqqt3/assignment_loss‚2.408732056617737 _runtime‚5668.669414792Ð\œHx’Ú‚(:‡ ÊïëÔ·¨¯Í gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³['Û‚(:ô ÊïëÔŸ¨ÇÎ% disk./.usageGB‚1180.3798370361328 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 13427.859375 disk.nvme0n1.in‚ 13427.859375 disk.sda3.in‚0 disk.sda.out‚ 199.9296875 disk.dm-1.out‚ 199.9296875" proc.memory.rssMB‚ 3053.1171875 network.sent‚ 1.0343681e+07 network.recv‚ 1.853771e+06) disk./.usagePercent‚79.21628126814582 disk.sda3.out‚ 199.9296875 disk.sda.in‚0$ disk.dm-0.out‚113885.82568359375 disk.dm-1.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚113885.82568359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚96.33793830871582' proc.memory.availableMB‚ 599.9921875* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.80078125' disk.nvme0n1.out‚113885.82568359375 disk.dm-0.in‚ 13427.859375 cpu‚25.38200715271996AK’~“Ü‚(:ˆ ÙïëÔµáèË gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý;5oÝ‚(:ä ÙïëÔȆñÌ disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.8046875' disk.nvme0c0n1.out‚114169.451171875" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 network.sent‚ 1.0438371e+07 network.recv‚ 1.859205e+06% disk./.usageGB‚1179.8936042785645 disk.sda.out‚ 200.74609375 disk.dm-1.out‚ 200.74609375 disk.sda2.out‚0) disk./.usagePercent‚79.18364977980895% disk.nvme0n1.out‚114169.451171875 disk.sda3.in‚0 disk.sda3.out‚ 200.74609375 disk.dm-1.in‚0 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 13427.859375 memory_percent‚ 96.337890625 disk.nvme0n1.in‚ 13427.859375 disk.dm-0.in‚ 13427.859375" disk.dm-0.out‚114169.451171875 disk.sda2.in‚0 proc.memory.availableMB‚600 cpu‚25.381328307999564 proc.cpu.threads‚17 disk.sda.in‚0•.ò“Þ‚(:ˆ èïëÔ±ÚôË gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚116.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W–~õ‚ß‚(:÷ èïëÔƒ›ÊÏ disk.drbd1260.out‚ 0.8046875' disk.nvme0c0n1.out‚114567.267578125 disk.sda.in‚0 disk.dm-0.in‚13581.91796875 disk.sda2.in‚0 disk.sda3.out‚ 201.33203125 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚114567.267578125 disk.dm-1.out‚ 201.33203125" proc.memory.rssMB‚ 3053.1171875 cpu‚25.380743506028825% disk.nvme0n1.out‚114567.267578125% disk./.usageGB‚1179.8701248168945 disk.sda2.out‚0" disk.nvme0n1.in‚13581.91796875 disk.sda.out‚ 201.33203125$ memory_percent‚96.33791446685791( proc.memory.availableMB‚ 599.99609375 proc.cpu.threads‚17 network.recv‚ 1.863592e+06$ disk.nvme0c0n1.in‚13581.91796875 disk.dm-1.in‚0* proc.memory.percent‚18.634748458862305 disk.sda3.in‚0 network.sent‚ 1.044433e+07) disk./.usagePercent‚79.18207405343568 disk.drbd1260.in‚0±Þù“à‚(:ˆ ÷ïëÔõÖËË gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?/âI{á‚(:ð ÷ïëÔ‡¿…Î* proc.memory.percent‚18.634748458862305 network.sent‚ 1.0450552e+07 disk.drbd1260.out‚ 0.80859375 disk.sda.out‚ 202.1953125! disk.dm-0.out‚114904.90234375( proc.memory.availableMB‚ 599.99609375 cpu‚25.380070168247165 proc.cpu.threads‚17) disk./.usagePercent‚79.13843552116417 disk.sda2.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 13585.4453125 disk.dm-1.out‚ 202.1953125" proc.memory.rssMB‚ 3053.1171875 disk.sda2.out‚0& disk.nvme0c0n1.out‚114904.90234375! disk.nvme0n1.in‚ 13585.4453125 disk.sda3.out‚ 202.1953125 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 network.recv‚ 1.868496e+06% disk./.usageGB‚1179.2198791503906$ disk.nvme0n1.out‚114904.90234375 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 13585.4453125$ memory_percent‚96.337914466857916^$2â- _wandbruntime‚5729  _runtime‚5729fH™&Åã¿ _runtime‚ 5729.48248787 _step‚337 & lr-AdamW/pg1‚0.0013996756400471329 & lr-AdamW/pg2‚0.0013996756400471329 trainer/global_step‚4149 $ _timestamp‚1.788540934171499e+09ôtN¥ÊäÄ & lr-AdamW/pg1‚0.0013996756400471329 & lr-AdamW/pg2‚0.0013996756400471329 trainer/global_step‚4149 $ _timestamp‚1.788540934171499e+09 _runtime‚ 5729.48248787 _step‚337ÑÍÝ4>2å- _wandbruntime‚5729  _runtime‚5729:O¹žö'æðO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385092231563776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104988012 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104988012 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748432137216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348998374396416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86599756 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390055454616064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353745878559232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86599758 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748432137216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191587768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353746806533632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25175187 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337475385853200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4730676216972 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4729768241968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51884018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25174547 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86599756 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105014356 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353745878559232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86599756 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105014356 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353745878559232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104988012 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batc!ßÍ4h_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748432137216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348998374396416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144Òä=À“è‚(:ˆ †ðëÔП¹Ê gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kW¾lé‚(:á †ðëÔöÇÍ$ memory_percent‚96.34006023406982( proc.memory.availableMB‚ 599.64453125* proc.memory.percent‚18.634748458862305 proc.cpu.threads‚17 disk.sda3.out‚ 202.8203125 network.sent‚ 1.0455592e+07 network.recv‚ 1.872178e+06) disk./.usagePercent‚79.13686184285119 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.80859375 disk.sda.out‚ 202.8203125# disk.nvme0c0n1.in‚ 13585.4453125! disk.nvme0c0n1.out‚ 115312.375! disk.nvme0n1.in‚ 13585.4453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 13585.4453125 disk.dm-0.out‚ 115312.375 disk.dm-1.out‚ 202.8203125 disk.nvme0n1.out‚ 115312.375 disk.sda3.in‚0% disk./.usageGB‚1179.1964302062988 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 3053.1171875 cpu‚25.379352737723345§–‚2ê- _wandbruntime‚5730  _runtime‚5730¢ Ô}'ëúM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77076449 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349079993482240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86619918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105039213 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86619916 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191659131 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25181008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385182457268224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4964385127424 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86619916 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337554403447744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349079975479296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚5730.486057417 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4149 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77077091 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390147101395968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337554420487104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885409351743903e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731793490458 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191659131 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749553610752iCF'ìÿM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191659131 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749553610752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77076449 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚349079993482240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86619918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105039213 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86619916 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191659131 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259000320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25181008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385182457268224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4964385127424 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86619916 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337554403447744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚349079975479296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚5730.486057417 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4149 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77077091 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390147101395968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337554420487104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235060224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885409351743903e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731793490458 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615Ó\Wr˜2í- _wandbruntime‚5730  _runtime‚5730‹D™KîÅ 0loss/FBt3/assignment_loss‚0.5504044329928107 loss/FBt4/assignment_loss‚0 1loss/SRqqt2/detection_loss‚2.0978355519473553 %loss/total_loss‚42.93236169736349 trainer/global_step‚4149 % _timestamp‚1.7885409351765714e+09 1loss/SRqqt1/detection_loss‚2.4764586002738387 .loss/FRt2/detection_loss‚2.651903137050826 1loss/SRqqt4/detection_loss‚1.2115370483560997 2loss/SRqqt1/assignment_loss‚2.0238211706832603 /loss/FRt3/detection_loss‚2.1820169610499933 epoch‚3 /loss/FRt2/assignment_loss‚2.812575504697602 /loss/FRt3/assignment_loss‚3.178428029691851 /loss/FRt1/detection_loss‚2.3447534974132265 /loss/FRt4/detection_loss‚1.6785918144803298 /loss/FBt2/detection_loss‚1.4514316430069365 /loss/FBt4/detection_loss‚0.2764425277709961 /loss/FRt1/assignment_loss‚3.243841044959568 2loss/SRqqt2/assignment_loss‚2.1368064810230862 0loss/FBt1/assignment_loss‚0.4289583796284126 /loss/FBt1/detection_loss‚2.4398603620736496 0loss/FRt4/assignment_loss‚2.0272976829294573 2loss/SRqqt3/assignment_loss‚2.3237281299772716 1loss/FBt2/assignment_loss‚0.42045877131960346 /loss/FBt3/detection_loss‚1.2686603692886622 _runtime‚5730.487424484 _step‚340 2loss/SRqqt4/assignment_loss‚0.9790164215659554 1loss/SRqqt3/detection_loss‚2.7275341351826987÷ ÓPïÊ /loss/FRt1/assignment_loss‚3.243841044959568 2loss/SRqqt2/assignment_loss‚2.1368064810230862 0loss/FBt1/assignment_loss‚0.4289583796284126 /loss/FBt1/detection_loss‚2.4398603620736496 0loss/FRt4/assignment_loss‚2.0272976829294573 2loss/SRqqt3/assignment_loss‚2.3237281299772716 1loss/FBt2/assignment_loss‚0.42045877131960346 /loss/FBt3/detection_loss‚1.2686603692886622 _runtime‚5730.487424484 _step‚340 2loss/SRqqt4/assignment_loss‚0.9790164215659554 1loss/SRqqt3/detection_loss‚2.7275341351826987 0loss/FBt3/assignment_loss‚0.5504044329928107 loss/FBt4/assignment_loss‚0 1loss/SRqqt2/detection_loss‚2.0978355519473553 %loss/total_loss‚42.93236169736349 trainer/global_step‚4149 % _timestamp‚1.7885409351765714e+09 1loss/SRqqt1/detection_loss‚2.4764586002738387 .loss/FRt2/detection_loss‚2.651903137050826 1loss/SRqqt4/detection_loss‚1.2115370483560997 2loss/SRqqt1/assignment_loss‚2.0238211706832603 /loss/FRt3/detection_loss‚2.1820169610499933 epoch‚3 /loss/FRt2/assignment_loss‚2.812575504697602 /loss/FRt3/assignment_loss‚3.178428029691851 /loss/FRt1/detection_loss‚2.3447534974132265 /loss/FRt4/detection_loss‚1.6785918144803298 /loss/FBt2/detection_loss‚1.4514316430069365 /loss/FBt4/detection_loss‚0.2764425277709961Ô_ªž‘ð‚(:† •ðëÔž§Í gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.63' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e¶=Þwñ‚(:ì •ðëÔÏ‘—Ð' proc.memory.availableMB‚ 599.7109375 cpu‚25.37868669791202 network.recv‚ 1.877521e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚0.8125 disk.sda3.in‚0 disk.sda1.in‚0$ disk.dm-0.out‚115523.63330078125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 13585.4453125 disk.sda1.out‚0 disk.dm-0.in‚ 13585.4453125 disk.dm-1.in‚0" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 network.sent‚ 1.055195e+07! disk.nvme0n1.in‚ 13585.4453125 disk.sda3.out‚ 203.59375 disk.sda.in‚0 disk.sda.out‚ 203.59375 disk.dm-1.out‚ 203.59375$ memory_percent‚96.33965492248535 proc.cpu.threads‚17) disk./.usagePercent‚79.13843372911143% disk./.usageGB‚1179.2198524475098 disk.sda2.out‚0) disk.nvme0c0n1.out‚115523.63330078125' disk.nvme0n1.out‚115523.63330078125»žò‚(:“ ¤ðëÔ‚™Ë gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.880399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨(¬‚ó‚(:÷ ¤ðëÔÝøßÌ disk.drbd1260.out‚0.8125 disk.sda.in‚0 disk.sda.out‚ 204.04296875 disk.sda1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1179.2198333740234# disk.nvme0c0n1.in‚ 13585.4453125) disk.nvme0c0n1.out‚115523.91845703125$ memory_percent‚96.33820056915283( proc.memory.availableMB‚ 599.94921875" proc.memory.rssMB‚ 3053.1171875 network.sent‚ 1.0651787e+07) disk./.usagePercent‚79.13843244907375' disk.nvme0n1.out‚115523.91845703125 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 13585.4453125 disk.dm-1.in‚0 disk.dm-1.out‚ 204.04296875 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 13585.4453125 disk.sda3.out‚ 204.04296875$ disk.dm-0.out‚115523.91845703125* proc.memory.percent‚18.634748458862305 cpu‚25.37799829117001 network.recv‚ 1.883641e+06 disk.drbd1260.in‚0o?ÿ…“ô‚(:ˆ ³ðëÔšüÕÎ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸5´Ærõ‚(:ç ³ðëÔÑœ Ò# disk./.usageGB‚1179.19633102417 disk.nvme0n1.in‚ 13750.71875' disk.nvme0n1.out‚115860.03662109375 disk.sda.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.888265e+06( disk./.usagePercent‚79.1368551866553$ disk.dm-0.out‚115860.03662109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚96.33815288543701( proc.memory.availableMB‚ 599.95703125 disk.dm-0.in‚ 13750.71875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.81640625! disk.nvme0c0n1.in‚ 13750.71875* proc.memory.percent‚18.634748458862305 network.sent‚ 1.0656109e+07 disk.dm-1.out‚204.8125) disk.nvme0c0n1.out‚115860.03662109375 disk.sda3.out‚204.8125 disk.sda.out‚204.8125 disk.sda1.in‚0" proc.memory.rssMB‚ 3053.1171875 cpu‚25.377447309085447E–Ë×2ö- _wandbruntime‚5789  _runtime‚5789PùÇ÷Á & lr-AdamW/pg2‚0.0013904871253885825 trainer/global_step‚4199 % _timestamp‚1.7885409942089415e+09 _runtime‚5789.520150881 _step‚341 & lr-AdamW/pg1‚0.0013904871253885825_â¹DÌøÆ _runtime‚5789.520150881 _step‚341 & lr-AdamW/pg1‚0.0013904871253885825 & lr-AdamW/pg2‚0.0013904871253885825 trainer/global_step‚4199 % _timestamp‚1.7885409942089415e+09Õ :Fl2ù- _wandbruntime‚5789  _runtime‚5789óDæúïO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚346211911077476 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106259258 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4803609590784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4804519562240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193867016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 trainer/global_step‚4199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start“ÍÜÜ/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353079328687616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353079310684672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5022199454208 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77952184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353079310684672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87607758 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87607758 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353079328687616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357882920275456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106232914 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52487266 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394625495846912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚342 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚5789.522308718 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357883848249856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357882920275456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341426282619760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786553472080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185ÀÖ¢uú'ûôO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193867016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 trainer/global_step‚4199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚353079328687616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚353079310684672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5022199454208 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77952184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚353079310684672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87607758 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87607758 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚353079328687616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357882920275456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106232914 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52487266 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394625495846912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚342 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 _runtime‚5789.522308718 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357883848249856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357882920275456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341426282619760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786553472080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚346211911077476 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106259258 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4803609590784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4804519562240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244333568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Örº–‘ü‚(:† ÂðëÔ×Ê·Ë gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚161.41' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø[(Œ~ý‚(:ó ÂðëÔœ½¡Í disk.dm-1.in‚0% disk./.usageGB‚1179.0897254943848 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.81640625! disk.nvme0c0n1.in‚ 13750.71875$ memory_percent‚96.33982181549072 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚116053.72412109375( proc.memory.availableMB‚ 599.68359375 cpu‚25.376736835517928 network.recv‚ 1.891233e+06) disk./.usagePercent‚79.12970080010342 disk.sda3.out‚ 205.5859375 disk.dm-0.in‚ 13750.71875 disk.sda2.out‚0 disk.nvme0n1.in‚ 13750.71875" proc.memory.rssMB‚ 3053.1171875* proc.memory.percent‚18.634748458862305 network.sent‚ 1.0660922e+07 disk.sda3.in‚0 disk.sda.out‚ 205.5859375$ disk.dm-0.out‚116053.72412109375 disk.dm-1.out‚ 205.5859375 disk.sda2.in‚0' disk.nvme0n1.out‚116053.72412109375 proc.cpu.threads‚17Ôq2þ- _wandbruntime‚5790  _runtime‚57909ãj'ÿúM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193885661 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786762770354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4804731051008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106284115 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357966588809216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25471271 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4787670745566 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52499333 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87627918 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚389693766430720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5023158559232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341505300214304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353160929770496 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193912033 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5023381036544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193912033 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4805641035776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346292045945298 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106257745 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚353160929770496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77969986 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚353160947773440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353160947773440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4805641035776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341505283174944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87627916 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394717147467264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885409954544795e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25470655 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389693742490624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87627918 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4804731051008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87627916 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856ÖÈï|¸€ÿM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚353160947773440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4805641035776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341505283174944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87627916 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394717147467264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 % _timestamp‚1.7885409954544795e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25470655 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389693742490624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87627918 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4804731051008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87627916 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193885661 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_traª|=ŽMin_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786762770354 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4804731051008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106284115 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357966588809216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25471271 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4787670745566 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52499333 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87627918 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚389693766430720 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5023158559232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341505300214304 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚353160929770496 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193912033 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5023381036544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193912033 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4805641035776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚346292045945298 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106257745 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚353160929770496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77969986 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚353160947773440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0×uO612- _wandbruntime‚5790  _runtime‚5790©µtP‚Ê 1loss/SRqqt4/detection_loss‚1.0720681778465708 % _timestamp‚1.7885409954574146e+09 -loss/FRt1/assignment_loss‚3.6594077830619 1loss/SRqqt2/assignment_loss‚1.566024968543878 2loss/SRqqt4/assignment_loss‚1.1082300008585055 1loss/SRqqt3/detection_loss‚2.4154771161930904 %loss/total_loss‚51.75196062119041 0loss/FBt3/assignment_loss‚0.5527051274360256 .loss/FBt1/detection_loss‚2.096413702800356 2loss/SRqqt1/assignment_loss‚2.2606664692100726 2loss/SRqqt3/assignment_loss‚2.0788097541247095 /loss/FRt3/detection_loss‚2.6284058685274263 _step‚344 0loss/FRt2/assignment_loss‚1.9229929012221258 0loss/SRqqt1/detection_loss‚3.131054015536057 /loss/FBt3/detection_loss‚2.2147497131448723 epoch‚3 /loss/FRt3/assignment_loss‚3.230536708998126 0loss/FBt1/assignment_loss‚0.3729352175763905 0loss/FBt2/assignment_loss‚0.2930856696785716 .loss/FRt1/detection_loss‚2.391196862972797 0loss/SRqqt2/detection_loss‚2.216061396094469 -loss/FBt2/detection_loss‚1.17098382311432 _runtime‚5790.768260875 0loss/FRt4/assignment_loss‚1.4102290272712708 /loss/FBt4/assignment_loss‚2.776482582092285 .loss/FRt2/detection_loss‚2.284740572845614 .loss/FBt4/detection_loss‚7.109681129455566 trainer/global_step‚4199 .loss/FRt4/detection_loss‚1.789022032585409ÜÙ*˜UƒÏ 0loss/SRqqt1/detection_loss‚3.131054015536057 /loss/FBt3/detection_loss‚2.2147497131448723 epoch‚3 /loss/FRt3/assignment_loss‚3.230536708998126 0loss/FBt1/assignment_loss‚0.3729352175763905 0loss/FBt2/assignment_loss‚0.2930856696785716 .loss/FRt1/detection_loss‚2.391196862972797 0loss/SRqqt2/detection_loss‚2.216061396094469 -loss/FBt2/detection_loss‚1.17098382311432 _runtime‚5790.768260875 0loss/FRt4/assignment_loss‚1.4102290272712708 /loss/FBt4/assignment_loss‚2.776482582092285 .loss/FRt2/detection_loss‚2.284740572845614 .loss/FBt4/detection_loss‚7.109681129455566 trainer/global_step‚4199 .loss/FRt4/detection_loss‚1.789022032585409 1loss/SRqqt4/detection_loss‚1.0720681778465708 % _timestamp‚1.7885409954574146e+09 -loss/FRt1/assignment_loss‚3.6594077830619 1loss/SRqqt2/assignment_loss‚1.566024968543878 2loss/SRqqt4/assignment_loss‚1.1082300008585055 1loss/SRqqt3/detection_loss‚2.4154771161930904 %loss/total_loss‚51.75196062119041 0loss/FBt3/assignment_loss‚0.5527051274360256 .loss/FBt1/detection_loss‚2.096413702800356 2loss/SRqqt1/assignment_loss‚2.2606664692100726 2loss/SRqqt3/assignment_loss‚2.0788097541247095 /loss/FRt3/detection_loss‚2.6284058685274263 _step‚344 0loss/FRt2/assignment_loss‚1.9229929012221258Ø»è<‘„‚(:† ÑðëÔÜøÅÌ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e-Ã{…‚(:ð ÑðëÔز»Î disk.dm-1.in‚0 disk.dm-1.out‚ 206.2265625 disk.sda2.in‚0) disk.nvme0c0n1.out‚116461.66748046875 disk.nvme0n1.in‚ 13750.71875 disk.sda.out‚ 206.2265625* proc.memory.percent‚18.634748458862305! disk.nvme0c0n1.in‚ 13750.71875 network.recv‚ 1.896666e+06% disk./.usageGB‚1179.1120567321777 disk.dm-0.in‚ 13750.71875 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8203125 disk.sda3.in‚0 disk.sda3.out‚ 206.2265625 disk.sda1.out‚0$ memory_percent‚96.33853435516357( proc.memory.availableMB‚ 599.89453125 network.sent‚ 1.0769478e+07$ disk.dm-0.out‚116461.66748046875' disk.nvme0n1.out‚116461.66748046875 disk.sda.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3053.1171875 cpu‚25.3761018937983 proc.cpu.threads‚17) disk./.usagePercent‚79.13119946820886Q\ê“ž†‚(:“ àðëÔÜï÷Ë gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.042' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.608399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sK~Ðs‡‚(:è àðëÔëóïÌ* proc.memory.percent‚18.634748458862305% disk./.usageGB‚1179.2775192260742 disk.sda.out‚206.8125 disk.sda1.out‚0 disk.dm-0.in‚ 13750.71875( proc.memory.availableMB‚ 599.89453125" proc.memory.rssMB‚ 3053.1171875 cpu‚25.37541042090293 proc.cpu.threads‚17) disk./.usagePercent‚79.14230379500653 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8203125) disk.nvme0c0n1.out‚116883.02685546875 disk.nvme0n1.in‚ 13750.71875 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚206.8125 network.sent‚ 1.0854435e+07 network.recv‚ 1.902598e+06 disk.sda3.out‚206.8125 disk.sda1.in‚0$ memory_percent‚96.33853435516357 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 13750.71875' disk.nvme0n1.out‚116883.02685546875 disk.sda3.in‚0$ disk.dm-0.out‚116883.02685546875 P1Æžˆ‚(:“ ïðëÔ°¬§Ì gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚118.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.799400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dçú‰‚(:ö ïðëÔ•¦ðÐ network.sent‚ 1.0860624e+07 disk.sda2.out‚0 disk.drbd1260.out‚ 0.82421875' disk.nvme0n1.out‚117059.91650390625 disk.sda3.out‚ 207.50390625$ disk.dm-0.out‚117059.91650390625 disk.dm-1.in‚0* proc.memory.percent‚18.634748458862305 network.recv‚ 1.907176e+06$ disk./.usageGB‚1179.276454925537 disk.nvme0n1.in‚ 13919.453125 disk.sda3.in‚0$ memory_percent‚96.34013175964355 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 207.50390625 disk.sda1.in‚0 disk.dm-0.in‚ 13919.453125 disk.dm-1.out‚ 207.50390625' proc.memory.availableMB‚ 599.6328125" proc.memory.rssMB‚ 3053.1171875) disk./.usagePercent‚79.14223236890449 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 13919.453125) disk.nvme0c0n1.out‚117059.91650390625 disk.sda1.out‚0 cpu‚25.37484313610439 proc.cpu.threads‚17ˆœÝ“Š‚(:ˆ þðëÔäýõÊ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚169.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ud~‹‚(:ó þðëÔíÞ—Ï network.recv‚ 1.910578e+06) disk./.usagePercent‚79.14065869059151% disk./.usageGB‚1179.2530059814453 disk.dm-1.out‚ 208.203125 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.82421875 disk.sda3.out‚ 208.203125 network.sent‚ 1.0865801e+07 disk.dm-1.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚117347.26416015625 disk.nvme0n1.in‚ 13919.453125 disk.sda.out‚ 208.203125 disk.sda1.out‚0$ disk.dm-0.out‚117347.26416015625" disk.nvme0c0n1.in‚ 13919.453125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 13919.453125$ memory_percent‚96.34010791778564" proc.memory.rssMB‚ 3053.1171875 proc.cpu.threads‚17' disk.nvme0n1.out‚117347.26416015625( proc.memory.availableMB‚ 599.63671875* proc.memory.percent‚18.634748458862305 cpu‚25.374243540473472K¦2Œ- _wandbruntime‚5850  _runtime‚5850®º2—Å¿ % _timestamp‚1.7885410554253583e+09 _runtime‚5850.736355882 _step‚345 % lr-AdamW/pg1‚0.001381184462372923 % lr-AdamW/pg2‚0.001381184462372923 trainer/global_step‚4249ß_®ÊŽÄ _runtime‚5850.736355882 _step‚345 % lr-AdamW/pg1‚0.001381184462372923 % lr-AdamW/pg2‚0.001381184462372923 trainer/global_step‚4249 % _timestamp‚1.7885410554253583e+09Ù?Ñ›I2- _wandbruntime‚5850  _runtime‚5850U$I‘ö'ðO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚350219593074148 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107504160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107477816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88615756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362019961991680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357160264975872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚5850.737887749 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196119918 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78846475 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚4249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88615756 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5080739440640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88615758 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚362020889966080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859697015808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107504160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4860606987264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357160264975872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚362020889966080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53090516 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5080951445504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25755959 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4842430727188 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345377162346960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4841522752184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196093572 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357160282978816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88615758 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345377145307600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107477816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399195777394176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25756594 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364Š]Ã(!‘õO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88615758 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚362020889966080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235944960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859697015808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107504160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4860606987264 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 _step‚346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚357160264975872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚362020889966080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53090516 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5080951445504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25755959 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4842430727188 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚345377162346960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4841522752184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196093572 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚357160282978816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88615758 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212004864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚345377145307600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107477816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399195777394176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25756594 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚350219593074148 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107504160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitorh³0ÔÓ.on_train_batch_start/active.small_pool.freed‚ 107477816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88615756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362019961991680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚357160264975872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚5850.737887749 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196119918 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78846475 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 trainer/global_step‚4249 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88615756 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5080739440640Ú`2’- _wandbruntime‚5852  _runtime‚5852@òà“'“úM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357241902064640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107502647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78864273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25761692 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚362103630525440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5082127655936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚347 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399286924503040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394205051653120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107502647 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357241884061696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107529017 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4842640025462 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4860818476032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4861728460800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88635916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4861728460800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860818476032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345456179941504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345456162902144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196138563 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350299727942178 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196164935 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53102581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚357241902064640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196164935 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚362102702537728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 % _timestamp‚1.7885410569180408e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107529017 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0÷BG'”ÿM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4843548000674 _runtime‚5852.229879675 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107502647 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚357241884061696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107529017 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4842640025462 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4860818476032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4861728460800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88635916 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4861728460800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860818476032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚345456179941504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚345456162902144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196138563 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350299727942178 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196164935 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53102581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚357241902064640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196164935 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚362102702537728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 % _timestamp‚1.7885410569180408e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107529017 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚357241902064640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107502647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78864273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25761692 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚362103630525440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5082127655936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚347 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399286924503040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394205051653120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4916694412288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398626087230976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885411149692106e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53693766 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89623758 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366157003707904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M=DeviceStatsMonitor.onHBL›_train_batch_start/active_bytes.all.peak‚ 14135738368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚354225425042092 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4898307982296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366157931682304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4915784440832 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108722718 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚354226350056456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198372820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26047038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246430720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108749062 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108722718 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361241219267072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366157003707904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361241237270016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5139429237248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398626111171072 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198346474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361241219267072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366157931682304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108749062 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108722718 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361241219267072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366157003707904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361241237270016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5139429237248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398626111171072 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198346474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361241219267072 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366157931682304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4916694412288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚398626087230976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 % _timestamp‚1.7885411149692106e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53693766 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89623758 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366157003707904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222490624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚354225425042092 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4898307982296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366157931682304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4915784440832 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108722718 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚354226350056456 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198372820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615Þé5ÿ2¦- _wandbruntime‚5911  _runtime‚5911 p :'§úM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198417837 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349407042629344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366240672241664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4898517280570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108747549 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79758651 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198391465 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4916905901056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89643916 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403857165184512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5140828308992 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4299 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89643916 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361322838352896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108773919 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79759284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361322856355840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198417837 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403856925058560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4917815885824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89643918 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361322856355840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361322838352896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89643918 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366239744253952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53705833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885411165262692e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349407059668704 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192„†;¸¨ÿM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403856925058560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4917815885824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240125952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89643918 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361322856355840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361322838352896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89643918 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366239744253952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53705833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 % _timestamp‚1.7885411165262692e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349407059668704 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198417837 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚349407042629344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366240672241664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4898517280570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 ;4DeviceStatsMonitor.on_train_batch_end/max_sp@yaŽðlit_size‚-1 _step‚351 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108747549 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79758651 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198391465 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4916905901056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89643916 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚403857165184512 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5140828308992 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4299 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89643916 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361322838352896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216185856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108773919 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79759284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361322856355840 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198417837 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856ßôE¸D2©- _wandbruntime‚5911  _runtime‚5911%÷†YFªÀ 1loss/SRqqt4/detection_loss‚1.2790511507274849 /loss/FBt2/detection_loss‚1.1038873560153522 .loss/FRt1/detection_loss‚2.410279282538051 0loss/FRt4/assignment_loss‚1.9112944861627974 0loss/FBt1/assignment_loss‚0.3145001325305518 0loss/SRqqt3/detection_loss‚2.261287828286489 1loss/FBt2/assignment_loss‚0.27415329010052675 /loss/FBt1/detection_loss‚2.3391588913357775 _runtime‚5911.840313934 0loss/FRt2/assignment_loss‚2.5072930693626403 1loss/SRqqt3/assignment_loss‚1.935424545046055 0loss/FRt1/assignment_loss‚3.5709736320105465 1loss/SRqqt1/assignment_loss‚1.972046154386857 .loss/FRt2/detection_loss‚2.535853919138511 0loss/SRqqt1/detection_loss‚3.129139409345739 trainer/global_step‚4299 /loss/FBt4/detection_loss‚0.9866714477539062 /loss/FRt3/assignment_loss‚4.096647830719643 2loss/SRqqt2/assignment_loss‚1.8327692105621098 2loss/SRqqt4/assignment_loss‚1.1966623379753865 1loss/SRqqt2/detection_loss‚2.2789110481739043 .loss/FRt3/detection_loss‚2.342681961716772 .loss/FRt4/detection_loss‚2.088566702913563 %loss/total_loss‚44.33359221422761 % _timestamp‚1.7885411165294936e+09 0loss/FBt3/assignment_loss‚0.5516884336608731 _step‚352 loss/FBt4/assignment_loss‚0 /loss/FBt3/detection_loss‚1.4146500937640667 epoch‚3«Ãº0K«Å 1loss/SRqqt3/assignment_loss‚1.935424545046055 0loss/FRt1/assignment_loss‚3.5709736320105465 1loss/SRqqt1/assignment_loss‚1.972046154386857 .loss/FRt2/detection_loss‚2.535853919138511 0loss/SRqqt1/detection_loss‚3.129139409345739 trainer/global_step‚4299 /loss/FBt4/detection_loss‚0.9866714477539062 /loss/FRt3/assignment_loss‚4.096647830719643 2loss/SRqqt2/assignment_loss‚1.8327692105621098 2loss/SRqqt4/assignment_loss‚1.1966623379753865 1loss/SRqqt2/detection_loss‚2.2789110481739043 .loss/FRt3/detection_loss‚2.342681961716772 .loss/FRt4/detection_loss‚2.088566702913563 %loss/total_loss‚44.33359221422761 % _timestamp‚1.7885411165294936e+09 0loss/FBt3/assignment_loss‚0.5516884336608731 _step‚352 loss/FBt4/assignment_loss‚0 /loss/FBt3/detection_loss‚1.4146500937640667 epoch‚3 1loss/SRqqt4/detection_loss‚1.2790511507274849 /loss/FBt2/detection_loss‚1.1038873560153522 .loss/FRt1/detection_loss‚2.410279282538051 0loss/FRt4/assignment_loss‚1.9112944861627974 0loss/FBt1/assignment_loss‚0.3145001325305518 0loss/SRqqt3/detection_loss‚2.261287828286489 1loss/FBt2/assignment_loss‚0.27415329010052675 /loss/FBt1/detection_loss‚2.3391588913357775 _runtime‚5911.840313934 0loss/FRt2/assignment_loss‚2.5072930693626403àѷГ¬‚(:ˆ ÉñëÔŸäßÎ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9346 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07l×~­‚(:ó ÉñëԛĭРdisk.sda3.out‚ 210.578125 disk.dm-0.in‚ 14051.7421875$ disk.dm-0.out‚118928.18115234375 network.sent‚ 1.0995884e+07# disk.nvme0c0n1.in‚ 14051.7421875) disk.nvme0c0n1.out‚118928.18115234375 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚96.34222984313965' proc.memory.availableMB‚ 599.2890625# proc.memory.rssMB‚ 3053.12109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.83984375 disk.sda.in‚0 disk.sda.out‚ 210.578125 disk.sda1.in‚0 disk.dm-1.out‚ 210.578125 cpu‚25.371250060182167' disk./.usagePercent‚79.159391785902' disk.nvme0n1.out‚118928.18115234375* proc.memory.percent‚18.634772300720215 proc.cpu.threads‚17 network.recv‚ 1.93217e+06% disk./.usageGB‚1179.5321426391602 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 14051.7421875 disk.sda3.in‚0œ»“®‚(:ˆ ØñëÔßêÍÍ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:ÅQ샯‚(:ø ØñëÔ¤‹ÃÏ network.recv‚ 1.93833e+06 disk.sda2.in‚0 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚119072.70849609375! disk.nvme0n1.in‚ 14051.7421875' disk.nvme0n1.out‚119072.70849609375 disk.sda3.out‚ 211.32421875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.83984375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 211.32421875$ memory_percent‚96.34075164794922% proc.memory.availableMB‚ 599.53125# proc.memory.rssMB‚ 3053.12109375 cpu‚25.37062010567238% disk./.usageGB‚1179.5086364746094 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 211.32421875 disk.dm-0.in‚ 14051.7421875* proc.memory.percent‚18.634772300720215 proc.cpu.threads‚17 network.sent‚ 1.1104338e+07) disk./.usagePercent‚79.15781426747601# disk.nvme0c0n1.in‚ 14051.7421875 disk.sda1.in‚0$ disk.dm-0.out‚119072.70849609375Ä1Þž°‚(:“ çñëÔÂÖòÊ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.585' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.717000000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?\¯âƒ±‚(:ø çñëÔÞ¡±Î disk.drbd1260.out‚0.84375) disk.nvme0c0n1.out‚119542.81689453125 disk.sda3.out‚ 211.94140625 disk.sda1.out‚0 network.sent‚ 1.1110573e+07 network.recv‚ 1.943315e+06 disk.sda2.out‚0 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚14112.11328125" disk.nvme0n1.in‚14112.11328125' disk.nvme0n1.out‚119542.81689453125 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 211.94140625 disk.sda1.in‚0$ memory_percent‚96.34075164794922% proc.memory.availableMB‚ 599.53125# proc.memory.rssMB‚ 3053.12109375 cpu‚25.3701200813214 proc.cpu.threads‚17) disk./.usagePercent‚79.17466033125163 disk.dm-0.in‚14112.11328125 disk.dm-1.out‚ 211.94140625 disk.sda2.in‚0* proc.memory.percent‚18.634772300720215% disk./.usageGB‚1179.7596549987793$ disk.dm-0.out‚119542.81689453125 disk.dm-1.in‚0¸Çx“²‚(:ˆ öñëÔâäÐË gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚127.756' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á´ß€³‚(:õ öñëÔÊ·—Í disk.sda1.out‚0$ disk.dm-0.out‚119773.77392578125 disk.dm-1.out‚ 212.55859375 disk.sda2.out‚0 disk.nvme0n1.in‚ 14237.265625' disk.nvme0n1.out‚119773.77392578125 disk.sda3.in‚0$ memory_percent‚96.34077548980713 network.sent‚ 1.111754e+07 disk.sda.out‚ 212.55859375 disk.dm-1.in‚0 disk.sda3.out‚ 212.55859375 cpu‚25.36941589389319 proc.cpu.threads‚17) disk./.usagePercent‚79.17616821562831 disk.sda2.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 599.52734375# proc.memory.rssMB‚ 3053.12109375* proc.memory.percent‚18.634772300720215 network.recv‚ 1.946896e+06% disk./.usageGB‚1179.7821235656738 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 14237.265625 disk.drbd1260.out‚0.84375" disk.nvme0c0n1.in‚ 14237.265625) disk.nvme0c0n1.out‚119773.77392578125KÏ,2´- _wandbruntime‚5971  _runtime‚5971Þæ„ǵÁ _step‚353 & lr-AdamW/pg1‚0.0013622444692987265 & lr-AdamW/pg2‚0.0013622444692987265 trainer/global_step‚4349 % _timestamp‚1.7885411766300042e+09 _runtime‚5971.941023615àÞ‘¢Ì¶Æ & lr-AdamW/pg2‚0.0013622444692987265 trainer/global_step‚4349 % _timestamp‚1.7885411766300042e+09 _runtime‚5971.941023615 _step‚353 & lr-AdamW/pg1‚0.0013622444692987265á?¸µH2·- _wandbruntime‚5971  _runtime‚5971ãÛºwö'¸ðO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4971871865856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358233107038764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54297016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5971.942226434 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370294045424128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198009519616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4349 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365322191561216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90631756 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953277262400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚365322173558272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4972781837312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370294045424128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353278921801360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4954185237404 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200599376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370294973398528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26338550 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109993964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80634928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90631756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109967620 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403137372453376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358232182024400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885411766311686e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54297018™trŸ‡!¹õO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90631756 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4953277262400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚365322173558272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4972781837312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370294045424128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353278921801360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4954185237404 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200599376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370294973398528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26338550 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109993964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80634928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90631756 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109967620 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403137372453376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358232182024400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 % _timestamp‚1.7885411766311686e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54297018 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4971871865856 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358233107038764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54297016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254819328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚5971.942226434 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚370294045424128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198009519616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4349 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365322191561216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249â[92º- _wandbruntime‚5973  _runtime‚5973Ã×6†'»úM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885411777022176e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54309081 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200644367 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200644367 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚4349 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90651916 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365403810647040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358313241906794 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26344391 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403227622097920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403227598157824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90651918 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365403792644096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370377713957888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200670739 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365403810647040 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26343766 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80653474 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5199174280704 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353357939395904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚355 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4973903310848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90651918 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110018821 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200670739 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80652847Ñá?'¼ÿM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403227622097920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403227598157824 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90651918 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365403792644096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚370377713957888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200670739 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365403810647040 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26343766 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80653474 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5199174280704 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353357939395904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 _step‚355 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4973903310848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90651918 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110018821 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200670739 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80652847 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246417408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 % _timestamp‚1.7885411777022176e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54309081 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200644367 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200644367 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222477312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 trainer/global_step‚4349 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90651916 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365403810647040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358313241906794 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26344391ãæšS2½- _wandbruntime‚5973  _runtime‚5973‹çòÜC¾½ 1loss/SRqqt2/assignment_loss‚2.406558322267873 2loss/SRqqt3/assignment_loss‚2.1811144753144336 .loss/FRt3/detection_loss‚2.678256196885362 1loss/SRqqt4/detection_loss‚0.9440483235549636 epoch‚3 /loss/FRt3/assignment_loss‚3.781235861778259 .loss/FBt3/detection_loss‚1.288759134709835 _step‚356 0loss/FRt1/assignment_loss‚3.2776714186919365 /loss/FRt1/detection_loss‚2.5740801974346765 .loss/FBt2/detection_loss‚1.296975985984318 trainer/global_step‚4349 /loss/FRt2/assignment_loss‚2.695882070355299 0loss/FBt3/assignment_loss‚0.6614741830296459 loss/FBt4/assignment_loss‚0 2loss/SRqqt1/assignment_loss‚1.8099542040573924 2loss/SRqqt4/assignment_loss‚1.4317672662285528 0loss/SRqqt2/detection_loss‚1.775226262637547 0loss/FBt4/detection_loss‚0.44802331924438477 $ _timestamp‚1.788541177706169e+09 /loss/FBt1/assignment_loss‚0.431306158239992 1loss/FBt2/assignment_loss‚0.26619141187184603 /loss/FRt2/detection_loss‚2.2764927163356687 0loss/SRqqt1/detection_loss‚3.085782926333578 0loss/SRqqt3/detection_loss‚2.504327443929819 0loss/FRt4/assignment_loss‚1.6367488633502614 .loss/FRt4/detection_loss‚2.012462405995889 .loss/FBt1/detection_loss‚2.042170069615046 %loss/total_loss‚43.50650921784658 _runtime‚5973.017094619â¼±ƒH¿ 0loss/FRt1/assignment_loss‚3.2776714186919365 /loss/FRt1/detection_loss‚2.5740801974346765 .loss/FBt2/detection_loss‚1.296975985984318 trainer/global_step‚4349 /loss/FRt2/assignment_loss‚2.695882070355299 0loss/FBt3/assignment_loss‚0.6614741830296459 loss/FBt4/assignment_loss‚0 2loss/SRqqt1/assignment_loss‚1.8099542040573924 2loss/SRqqt4/assignment_loss‚1.4317672662285528 0loss/SRqqt2/detection_loss‚1.775226262637547 0loss/FBt4/detection_loss‚0.44802331924438477 $ _timestamp‚1.788541177706169e+09 /loss/FBt1/assignment_loss‚0.431306158239992 1loss/FBt2/assignment_loss‚0.26619141187184603 /loss/FRt2/detection_loss‚2.2764927163356687 0loss/SRqqt1/detection_loss‚3.085782926333578 0loss/SRqqt3/detection_loss‚2.504327443929819 0loss/FRt4/assignment_loss‚1.6367488633502614 .loss/FRt4/detection_loss‚2.012462405995889 .loss/FBt1/detection_loss‚2.042170069615046 %loss/total_loss‚43.50650921784658 _runtime‚5973.017094619 1loss/SRqqt2/assignment_loss‚2.406558322267873 2loss/SRqqt3/assignment_loss‚2.1811144753144336 .loss/FRt3/detection_loss‚2.678256196885362 1loss/SRqqt4/detection_loss‚0.9440483235549636 epoch‚3 /loss/FRt3/assignment_loss‚3.781235861778259 .loss/FBt3/detection_loss‚1.288759134709835 _step‚356ä:·ú¸žÀ‚(:“ …òëÔ¸Î¥Ø gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚151.663' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.332600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê;…Á‚(:ú …òëÔð™•Ü disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 14237.265625) disk.nvme0c0n1.out‚120119.65087890625' disk.nvme0n1.out‚120119.65087890625 disk.sda3.out‚ 213.06640625# proc.memory.rssMB‚ 3053.12109375* proc.memory.percent‚18.634772300720215) disk./.usagePercent‚79.17616693559063 disk.sda1.in‚0 disk.dm-0.in‚ 14237.265625 disk.drbd1260.out‚ 0.84765625$ memory_percent‚96.34096622467041( proc.memory.availableMB‚ 599.49609375 proc.cpu.threads‚17% disk./.usageGB‚1179.7821044921875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.368866669233327 network.sent‚ 1.1120726e+07 network.recv‚ 1.950363e+06 disk.sda.out‚ 213.06640625 disk.sda1.out‚0$ disk.dm-0.out‚120119.65087890625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 14237.265625 disk.sda3.in‚0 disk.dm-1.out‚ 213.06640625é’³—‚(:’ ”òëÔçÌ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚168.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.772000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ®B¼zÂ(:ï ”òëÔçŸÇÎ$ disk.dm-0.out‚120330.72900390625 disk.dm-1.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚120330.72900390625 disk.sda1.in‚0 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚120330.72900390625 disk.sda3.in‚0 disk.sda3.out‚ 213.48828125 disk.sda.out‚ 213.48828125 proc.memory.availableMB‚599# proc.memory.rssMB‚ 3053.12109375 cpu‚25.368191589235586% disk./.usageGB‚1179.7820663452148 disk.dm-0.in‚ 14237.265625 disk.sda2.in‚0 disk.sda1.out‚0" memory_percent‚96.343994140625 proc.cpu.threads‚17) disk./.usagePercent‚79.17616437551528 disk.dm-1.out‚ 213.48828125 disk.drbd1260.out‚ 0.84765625 disk.nvme0n1.in‚ 14237.265625 disk.sda.in‚0* proc.memory.percent‚18.634772300720215" disk.nvme0c0n1.in‚ 14237.265625 network.sent‚ 1.1229214e+07 network.recv‚ 1.956508e+06Ð+÷MžÄ‚(:“ £òëÔ’Ä­Ë gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.369' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.273799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä•+p~Å‚(:ó £òëÔð´ÑÍ( proc.memory.availableMB‚ 599.49609375# proc.memory.rssMB‚ 3053.12109375" disk.nvme0c0n1.in‚ 14237.265625 disk.nvme0n1.in‚ 14237.265625 disk.sda3.out‚ 214.20703125 disk.sda.out‚ 214.20703125$ memory_percent‚96.34096622467041* proc.memory.percent‚18.634772300720215 network.recv‚ 1.961165e+06% disk./.usageGB‚1179.7586364746094 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 214.20703125 cpu‚25.367701063406482) disk./.usagePercent‚79.17459197723998 disk.sda2.in‚0 disk.drbd1260.out‚ 0.8515625% disk.nvme0n1.out‚120556.458984375 disk.sda1.in‚0" disk.dm-0.out‚120556.458984375 proc.cpu.threads‚17 network.sent‚ 1.1235532e+07 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚120556.458984375 disk.sda.in‚0 disk.dm-0.in‚ 14237.265625 disk.dm-1.in‚0X–EžÆ‚(:“ ²òëÔ”¬ºÌ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.786' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.357200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹duÇ‚(:ê ²òëÔú×îÑ disk.dm-1.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.634772300720215 proc.cpu.threads‚17 disk.sda3.out‚ 214.703125 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8515625# proc.memory.rssMB‚ 3053.12109375 cpu‚25.36712768763266 network.sent‚ 1.1242498e+07 network.recv‚ 1.964391e+06% disk./.usageGB‚1180.2586364746094 disk.sda.out‚ 214.703125" disk.dm-0.out‚121010.294921875 disk.sda2.out‚0# memory_percent‚96.3409423828125! proc.memory.availableMB‚599.5) disk./.usagePercent‚79.20814739676794 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚14392.32421875 disk.dm-1.out‚ 214.703125$ disk.nvme0c0n1.in‚14392.32421875' disk.nvme0c0n1.out‚121010.294921875" disk.nvme0n1.in‚14392.32421875% disk.nvme0n1.out‚121010.294921875˜zf2È- _wandbruntime‚6034  _runtime‚6034 "\:ÆÉÀ trainer/global_step‚4399 $ _timestamp‚1.788541239264955e+09 _runtime‚6034.576022089 _step‚357 & lr-AdamW/pg1‚0.0013526110863063582 & lr-AdamW/pg2‚0.0013526110863063582}Ñ"6ËÊÅ & lr-AdamW/pg2‚0.0013526110863063582 trainer/global_step‚4399 $ _timestamp‚1.788541239264955e+09 _runtime‚6034.576022089 _step‚357 & lr-AdamW/pg1‚0.0013526110863063582åìpã,2Ë- _wandbruntime‚6034  _runtime‚6034yÏ°¹® ÌðO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374431087140352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5009154517508 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81528761 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362238939006708 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111238866 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81529383 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91639758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26628495 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357229784489200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5010062492512 _step‚358 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.alloÓüŸHcated‚ 5028869262336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91639758 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5256683389952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5028869262336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54900268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885412392666554e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362239864021072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374431087140352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26629115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407648657675776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111212522 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5027959290880 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412905112283648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5027959290880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54900266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111238866 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374432015114752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357229801528560 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374432015114752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111212522 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5028869262336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54900268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885412392666554e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚362239864021072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374431087140352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26629115 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407648657675776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111212522 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5027959290880 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412905112283648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5027959290880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54900266 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111238866 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374432015114752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357229801528560 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374432015114752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111212522 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374431087140352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5009154517508 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81528761 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362238939006708 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111238866 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81529383 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91639758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228782080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26628495 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357229784489200 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5010062492512 _step‚358æÐþˆ×2Î- _wandbruntime‚6036  _runtime‚6036–-p”'ÏúM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369484746935296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26634878 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362319998889102 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369484764938240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111263723 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5029990735872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91659916 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5029990735872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5029080751104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81546585 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5257839696896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111237353 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202923641 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412996747017216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257600442368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5011179765998 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885412407004044e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111237353 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5029080751104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374514755674112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407738907320320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91659918 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412996483822592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369484746935296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362319073874530 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111263723 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5010271790786 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374513827686400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202897269 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202897269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91659918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54912333 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚6036.013231015 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91659916 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 239254528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357308819123104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26634878 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362319998889102 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369484764938240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111263723 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5029990735872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91659916 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5029990735872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5029080751104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81546585 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.all¹[ÈÈocated‚ 5257839696896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111237353 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202923641 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412996747017216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257600442368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5011179765998 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 % _timestamp‚1.7885412407004044e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111237353 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5029080751104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374514755674112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407738907320320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91659918 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263194624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412996483822592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚369484746935296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚626 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362319073874530 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111263723 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5010271790786 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374513827686400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202897269 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202897269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91659918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54912333 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _runtime‚6036.013231015 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91659916 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248çHŽºô2Ñ- _wandbruntime‚6036  _runtime‚6036ÖÃssYÒÓ 2loss/SRqqt1/assignment_loss‚2.0827117908000945 1loss/FBt2/assignment_loss‚0.20372755570895187 /loss/FBt3/detection_loss‚1.8468974191566994 0loss/FRt2/assignment_loss‚2.3911139273643496 % _timestamp‚1.7885412407043326e+09 _step‚360 0loss/FRt1/assignment_loss‚2.5599613611291097 1loss/SRqqt3/assignment_loss‚2.350281608778806 0loss/SRqqt3/detection_loss‚2.694518498011998 %loss/total_loss‚52.18224081686827 _runtime‚6036.015306155 0loss/FRt4/assignment_loss‚2.1221225283764027 2loss/SRqqt4/assignment_loss‚1.2384515568712908 trainer/global_step‚4399 /loss/FRt2/detection_loss‚3.1130018961429595 2loss/SRqqt2/assignment_loss‚1.3676012098789214 1loss/FBt1/assignment_loss‚0.38325193471779734 /loss/FBt4/assignment_loss‚2.656944751739502 /loss/FRt3/detection_loss‚2.1647313395664334 1loss/SRqqt4/detection_loss‚1.1105125823379618 .loss/FBt4/detection_loss‚6.940337181091309 /loss/FRt3/assignment_loss‚3.776159420991555 .loss/FRt1/detection_loss‚2.424229822507719 /loss/FBt1/detection_loss‚2.1757117390632628 /loss/FBt2/detection_loss‚1.3592695587298327 epoch‚3 0loss/FBt3/assignment_loss‚0.5877628398340464 0loss/SRqqt1/detection_loss‚2.609042453765869 .loss/FRt4/detection_loss‚1.702984807116014 1loss/SRqqt2/detection_loss‚2.3209130331873893Óþ4Ñ^ÓØ /loss/FRt3/detection_loss‚2.1647313395664334 1loss/SRqqt4/detection_loss‚1.1105125823379618 .loss/FBt4/detection_loss‚6.940337181091309 /loss/FRt3/assignment_loss‚3.776159420991555 .loss/FRt1/detection_loss‚2.424229822507719 /loss/FBt1/detection_loss‚2.1757117390632628 /loss/FBt2/detection_loss‚1.3592695587298327 epoch‚3 0loss/FBt3/assignment_loss‚0.5877628398340464 0loss/SRqqt1/detection_loss‚2.609042453765869 .loss/FRt4/detection_loss‚1.702984807116014 1loss/SRqqt2/detection_loss‚2.3209130331873893 2loss/SRqqt1/assignment_loss‚2.0827117908000945 1loss/FBt2/assignment_loss‚0.20372755570895187 /loss/FBt3/detection_loss‚1.8468974191566994 0loss/FRt2/assignment_loss‚2.3911139273643496 % _timestamp‚1.7885412407043326e+09 _step‚360 0loss/FRt1/assignment_loss‚2.5599613611291097 1loss/SRqqt3/assignment_loss‚2.350281608778806 0loss/SRqqt3/detection_loss‚2.694518498011998 %loss/total_loss‚52.18224081686827 _runtime‚6036.015306155 0loss/FRt4/assignment_loss‚2.1221225283764027 2loss/SRqqt4/assignment_loss‚1.2384515568712908 trainer/global_step‚4399 /loss/FRt2/detection_loss‚3.1130018961429595 2loss/SRqqt2/assignment_loss‚1.3676012098789214 1loss/FBt1/assignment_loss‚0.38325193471779734 /loss/FBt4/assignment_loss‚2.656944751739502èU-Hš“Ô‚(:ˆ ÁòëÔåÕË gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.436' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âSTÛvÕ‚(:ë ÁòëÔ’‡Î$ disk./.usageGB‚1180.258617401123 disk.sda.out‚ 215.11328125 disk.sda1.out‚0 disk.dm-1.out‚ 215.11328125* proc.memory.percent‚18.634772300720215 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda3.out‚ 215.11328125 disk.sda.in‚0 disk.dm-0.in‚14392.32421875$ memory_percent‚96.34437561035156# proc.memory.rssMB‚ 3053.12109375 network.recv‚ 1.968088e+06 disk.drbd1260.out‚ 0.85546875" disk.nvme0n1.in‚14392.32421875" disk.nvme0n1.out‚ 121401.796875 disk.sda3.in‚0 disk.dm-0.out‚ 121401.796875$ proc.memory.availableMB‚598.9375 network.sent‚ 1.1258767e+07$ disk.nvme0c0n1.in‚14392.32421875$ disk.nvme0c0n1.out‚ 121401.796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.36654313692584) disk./.usagePercent‚79.20814611673026DPgk“Ö‚(:ˆ ÐòëÔý€Í gpu.0.gpu‚43.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨šµ:€×‚(:õ ÐòëÔ¥ŒöÐ disk.drbd1260.out‚ 0.85546875$ disk.nvme0c0n1.in‚14392.32421875& disk.nvme0c0n1.out‚121747.42578125$ memory_percent‚96.34134769439697# proc.memory.rssMB‚ 3053.12109375" disk.nvme0n1.in‚14392.32421875 disk.sda3.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.634772300720215 proc.cpu.threads‚17 network.sent‚ 1.1354181e+07 network.recv‚ 1.974219e+06$ disk./.usageGB‚1179.891544342041$ disk.nvme0n1.out‚121747.42578125 disk.sda3.out‚ 215.73046875 disk.sda.in‚0 disk.sda.out‚ 215.73046875 disk.sda1.in‚0 disk.dm-0.in‚14392.32421875 disk.dm-1.in‚0 disk.dm-1.out‚ 215.73046875 disk.sda2.out‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 599.43359375 cpu‚25.36598732394257) disk./.usagePercent‚79.18351153574046 disk.sda1.out‚0! disk.dm-0.out‚121747.42578125•T¼“Ø‚(:ˆ ßòëÔ­‘Ì gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G}Ù‚(:ô ßòëÔéÆ–Ð network.sent‚ 1.1360369e+07) disk./.usagePercent‚79.18351537585346 disk.sda2.in‚0$ disk.nvme0c0n1.in‚14392.32421875' proc.memory.availableMB‚ 599.1796875* proc.memory.percent‚18.634796142578125" disk./.usageGB‚1179.8916015625& disk.nvme0n1.out‚122006.7705078125 disk.sda3.in‚0 disk.sda3.out‚ 216.09765625 disk.sda.out‚ 216.09765625 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚96.34289741516113 disk.sda2.out‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚122006.7705078125 disk.sda1.out‚0 disk.dm-0.in‚14392.32421875# disk.dm-0.out‚122006.7705078125 disk.dm-1.out‚ 216.09765625 proc.cpu.threads‚17 network.recv‚ 1.978664e+06 disk.drbd1260.out‚ 0.86328125" disk.nvme0n1.in‚14392.32421875 disk.sda.in‚0 proc.memory.rssMB‚3053.125 cpu‚25.365382860430913eC Õ“Ú‚(:ˆ îòëÔ¹¢øÌ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹/roÛ‚(:ä îòëÔÆ»³Ð( proc.memory.availableMB‚ 599.41796875 proc.memory.rssMB‚3053.125 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.634796142578125 cpu‚25.364813788181518 proc.cpu.threads‚17 network.recv‚ 1.98203e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.86328125 disk.nvme0n1.in‚14546.75 disk.sda.out‚ 216.76953125 disk.dm-0.in‚14546.75$ memory_percent‚96.34144306182861( disk./.usagePercent‚79.1819416975405% disk./.usageGB‚1179.8681526184082( disk.nvme0c0n1.out‚122208.2275390625& disk.nvme0n1.out‚122208.2275390625 disk.sda1.in‚0# disk.dm-0.out‚122208.2275390625 disk.dm-1.in‚0 disk.dm-1.out‚ 216.76953125 network.sent‚ 1.1365385e+07 disk.nvme0c0n1.in‚14546.75 disk.sda3.in‚0 disk.sda3.out‚ 216.76953125 disk.sda.in‚0 disk.sda2.out‚0;†‡2Ü- _wandbruntime‚6095  _runtime‚6095{ ÝÅÝ¿ & lr-AdamW/pg1‚0.0013428714482666558 & lr-AdamW/pg2‚0.0013428714482666558 trainer/global_step‚4449 # _timestamp‚1.78854130036723e+09 _runtime‚6095.678185491 _step‚361.L$¡ÊÞÄ _runtime‚6095.678185491 _step‚361 & lr-AdamW/pg1‚0.0013428714482666558 & lr-AdamW/pg2‚0.0013428714482666558 trainer/global_step‚4449 # _timestamp‚1.78854130036723e+09éOñÍ2ß- _wandbruntime‚6095  _runtime‚6095¼†Çjö'àðO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚362 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5084046715904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚378569056830976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5084956687360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55503518 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92647756 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5084046715904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417475217069056 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378568128856576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82421896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373484100143616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373484100143616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26918380 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5065031772616 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112483768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361180681255760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205105180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92647758 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚6095.679309739 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚378568128856576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82421896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚373484100143616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚373484100143616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26918380 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5065031772616 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112483768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361180681255760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205105180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92647758 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 _runtime‚6095.679309739 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 _step‚362 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5084046715904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚378569056830976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5084956687360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235073536 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259013632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55503518 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92647756 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5084046715904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417475217069056 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5067057021106 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92667916 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205176543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373565719229440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5316412580352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112508625 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205150171 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82439713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55515581 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚363 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 _runtime‚6096.961706775 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5085168176128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112508625 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417566605123072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412250192542720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412250168602624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417566348219904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26924132 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92667918 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366326755871410 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378651797390336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5086078160896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92667916 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5066149045894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55515583 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885413016502256e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205150171 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5086078160896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5316179617280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373565719229440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92667918 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚361259681810944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205176543 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112482255 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚378650869402624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82440358 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82439713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55515581 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚363 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 _runtime‚6096.961706775 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5085168176128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112508625 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417566605123072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412250192542720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412250168602624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417566348219904 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26924132 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92667918 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366326755871410 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚378651797390336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5086078160896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92667916 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256903168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5066149045894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55515583 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 % _timestamp‚1.7885413016502256e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205150171 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5086078160896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5316179617280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373565719229440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92667918 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚361259681810944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205176543 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112482255 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚378650869402624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82440358 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5067057021106 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92667916 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205176543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373565719229440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5316412580352뀺!í2å- _wandbruntime‚6096  _runtime‚6096YeâVæÐ .loss/FRt3/detection_loss‚2.463167875515986 trainer/global_step‚4449 0loss/FRt1/assignment_loss‚2.7608172044157984 2loss/SRqqt1/assignment_loss‚1.6301934116830428 0loss/FBt3/assignment_loss‚0.6836527895041408 .loss/FBt4/detection_loss‚7.288441181182861 0loss/FRt4/assignment_loss‚2.4482895698812275 /loss/FRt1/detection_loss‚2.5097784457728265 -loss/FBt1/detection_loss‚2.44454162371786 %loss/total_loss‚51.97891265534817 epoch‚3 _step‚364 2loss/SRqqt4/assignment_loss‚1.3770442584768758 /loss/FRt4/detection_loss‚1.8693589764060798 /loss/FBt2/detection_loss‚1.1023199043389578 0loss/FRt2/assignment_loss‚1.8872281764492844 2loss/SRqqt2/assignment_loss‚1.6856274373001523 2loss/SRqqt3/assignment_loss‚1.8895127284526825 0loss/FBt4/assignment_loss‚3.0287487506866455 0loss/SRqqt4/detection_loss‚1.182122375576868 0loss/FRt3/assignment_loss‚3.4729291717211406 0loss/FBt1/assignment_loss‚0.4741132617167942 1loss/SRqqt1/detection_loss‚2.9487440635760627 -loss/FBt3/detection_loss‚1.51924117654562 % _timestamp‚1.7885413016523988e+09 /loss/FRt2/detection_loss‚2.1530974128029565 0loss/FBt2/assignment_loss‚0.2093855839228588 0loss/SRqqt2/detection_loss‚2.101736236501623 0loss/SRqqt3/detection_loss‚2.848821039199829 _runtime‚6096.963195691ΔE¤[çÕ % _timestamp‚1.7885413016523988e+09 /loss/FRt2/detection_loss‚2.1530974128029565 0loss/FBt2/assignment_loss‚0.2093855839228588 0loss/SRqqt2/detection_loss‚2.101736236501623 0loss/SRqqt3/detection_loss‚2.848821039199829 _runtime‚6096.963195691 .loss/FRt3/detection_loss‚2.463167875515986 trainer/global_step‚4449 0loss/FRt1/assignment_loss‚2.7608172044157984 2loss/SRqqt1/assignment_loss‚1.6301934116830428 0loss/FBt3/assignment_loss‚0.6836527895041408 .loss/FBt4/detection_loss‚7.288441181182861 0loss/FRt4/assignment_loss‚2.4482895698812275 /loss/FRt1/detection_loss‚2.5097784457728265 -loss/FBt1/detection_loss‚2.44454162371786 %loss/total_loss‚51.97891265534817 epoch‚3 _step‚364 2loss/SRqqt4/assignment_loss‚1.3770442584768758 /loss/FRt4/detection_loss‚1.8693589764060798 /loss/FBt2/detection_loss‚1.1023199043389578 0loss/FRt2/assignment_loss‚1.8872281764492844 2loss/SRqqt2/assignment_loss‚1.6856274373001523 2loss/SRqqt3/assignment_loss‚1.8895127284526825 0loss/FBt4/assignment_loss‚3.0287487506866455 0loss/SRqqt4/detection_loss‚1.182122375576868 0loss/FRt3/assignment_loss‚3.4729291717211406 0loss/FBt1/assignment_loss‚0.4741132617167942 1loss/SRqqt1/detection_loss‚2.9487440635760627 -loss/FBt3/detection_loss‚1.51924117654562ì;æ´“è‚(:ˆ ýòëÔÏ„¥Î gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.531' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²vµvé‚(:ë ýòëÔØì·Ò disk.nvme0c0n1.in‚14546.75' disk.nvme0n1.out‚122550.21142578125# proc.memory.rssMB‚ 3053.12890625 cpu‚25.364231975783813 network.recv‚ 1.984768e+06 disk.drbd1260.in‚0 disk.nvme0n1.in‚14546.75 disk.sda3.out‚ 217.10546875 disk.sda1.in‚0 disk.dm-0.in‚14546.75* proc.memory.percent‚18.634819984436035 proc.cpu.threads‚17% disk./.usageGB‚1179.8686447143555) disk.nvme0c0n1.out‚122550.21142578125 disk.sda3.in‚0 disk.sda.out‚ 217.10546875$ disk.dm-0.out‚122550.21142578125 disk.dm-1.in‚0 disk.dm-1.out‚ 217.10546875$ memory_percent‚96.34160995483398) disk./.usagePercent‚79.18197472251241 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.8671875 disk.sda.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 599.390625 network.sent‚ 1.1369636e+07,Ëžê‚(:“ ŒóëÔò¢÷Ì gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚123.175' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.634999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Çq£vë‚(:ë ŒóëÔþÑ% disk./.usageGB‚1179.8686256408691$ disk.dm-0.out‚122967.93798828125 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.nvme0c0n1.in‚14546.75 disk.sda3.out‚ 217.70703125# proc.memory.rssMB‚ 3053.12890625 disk.dm-1.in‚0) disk.nvme0c0n1.out‚122967.93798828125 disk.sda3.in‚0$ memory_percent‚96.34318351745605 cpu‚25.36375589970803 proc.cpu.threads‚17) disk./.usagePercent‚79.18197344247474 disk.sda1.in‚0 disk.dm-0.in‚14546.75 disk.dm-1.out‚ 217.70703125 disk.drbd1260.out‚ 0.8671875' proc.memory.availableMB‚ 599.1328125* proc.memory.percent‚18.634819984436035 network.recv‚ 1.991156e+06 disk.sda1.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚14546.75' disk.nvme0n1.out‚122967.93798828125 disk.sda.in‚0 disk.sda.out‚ 217.70703125 network.sent‚ 1.1478365e+07A_WR“ì‚(:ˆ ›óëÔêÃÌ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚125.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á’Ùoí‚(:ä ›óëÔ•ÒÕÐ disk.dm-1.in‚0) disk./.usagePercent‚79.18353073630551 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-0.in‚14546.75! disk.dm-0.out‚123463.14453125$ memory_percent‚96.34296894073486# proc.memory.rssMB‚ 3053.12890625* proc.memory.percent‚18.634819984436035 disk.nvme0c0n1.in‚14546.75$ disk.nvme0n1.out‚123463.14453125 disk.sda3.in‚0 disk.sda3.out‚ 218.14453125 disk.sda.in‚0 disk.dm-1.out‚ 218.14453125 cpu‚25.363127115886744 proc.cpu.threads‚17 network.recv‚ 1.995862e+06 disk.nvme0n1.in‚14546.75 disk.sda.out‚ 218.14453125( proc.memory.availableMB‚ 599.16796875 network.sent‚ 1.1484595e+07$ disk./.usageGB‚1179.891830444336 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.87109375& disk.nvme0c0n1.out‚123463.14453125 disk.sda1.in‚0‘TAf“î‚(:ˆ ªóëÔÀ‡‹Ì gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Ùï{ï‚(:ð ªóëÔûÓÿÌ( proc.memory.availableMB‚ 599.16796875# proc.memory.rssMB‚ 3053.12890625 network.sent‚ 1.1491533e+07 disk.dm-0.in‚ 14704.1640625! disk.dm-0.out‚123573.97265625 disk.dm-1.out‚ 218.6796875# disk.nvme0c0n1.in‚ 14704.1640625! disk.nvme0n1.in‚ 14704.1640625 disk.sda3.out‚ 218.6796875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.634819984436035 cpu‚25.362642396717625% disk./.usageGB‚1178.0434494018555 disk.drbd1260.in‚0$ disk.nvme0n1.out‚123573.97265625 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.34296894073486 network.recv‚ 1.99916e+06) disk./.usagePercent‚79.05948433364966 disk.sda2.out‚0 disk.drbd1260.out‚ 0.87109375& disk.nvme0c0n1.out‚123573.97265625 disk.sda.out‚ 218.6796875'¯Á2ð- _wandbruntime‚6155  _runtime‚6155EðDÈÇñÁ % _timestamp‚1.7885413606019664e+09 _runtime‚6155.913013973 _step‚365 & lr-AdamW/pg1‚0.0013330275849055882 & lr-AdamW/pg2‚0.0013330275849055882 trainer/global_step‚4499åHnjÌòÆ _runtime‚6155.913013973 _step‚365 & lr-AdamW/pg1‚0.0013330275849055882 & lr-AdamW/pg2‚0.0013330275849055882 trainer/global_step‚4499 % _timestamp‚1.7885413606019664e+09íSØX[2ó- _wandbruntime‚6155  _runtime‚6155¾û¤ ôðO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377565054434816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382705170572800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377565036431872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382705170572800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 _runtime‚6155.914787876 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207358082 YEDevicŸÞçmØeStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382706098547200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365131543943600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382706098547200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5140134140928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83315900 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93655758 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56106768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56106766 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5141044112384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5141044112384 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370252452971324 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚366 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27209132 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5373711666688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5140134140928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370253377985688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5121817002728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5120909027724 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93655756 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377565036431872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83315276 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422044939787264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416671252060672 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144–DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370252452971324 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚366 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27209132 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5373711666688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5140134140928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370253377985688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5121817002728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5120909027724 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93655756 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377565036431872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83315276 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422044939787264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416671252060672 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377565054434816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382705170572800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377565036431872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382705170572800 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 _runtime‚6155.914787876 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207358082 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382706098547200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365131543943600 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382706098547200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5140134140928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83315900 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93655758 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56106768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56106766 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5141044112384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5141044112384 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5122934276214 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365210578577504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377646673520640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382788839106560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370333512853718 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83333141 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370332587839146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5142165585920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93675916 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113727157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27214928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5374873967104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377646673520640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27214310 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5141255601152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚6157.594005974 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56118831 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416761453825024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365210561538144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5142165585920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5122026301002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382787911118848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207429445ïÎœØøÿM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5142165585920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93675916 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113727157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27214928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5374873967104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377646673520640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27214310 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5141255601152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚6157.594005974 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56118831 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416761453825024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 232963072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K;DeviceStatsMonitor.ÀTë¡-on_train_batch_end/active_bytes.all.peak‚ 14135738368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚365210561538144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5142165585920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5122026301002 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚382787911118848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207429445 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93675916 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚4499 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207403073 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113727157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113753527 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5122934276214 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚365210578577504 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377646673520640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382788839106560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370333512853718 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83333141 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚370332587839146 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5432187062784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚6214.627070484 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386842212289024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369082423670800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381646008726016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114973572 _step‚370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56710018 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386842212289024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94663758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209610984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114947228 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421182513342976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94663758 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5196221565952¼³IžD"‰õO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5196221565952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381646008726016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94663756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256916480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚369082440710160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421182537283072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94663756 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56710016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 232976384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374260134967996 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5177694257836 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386843140263424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5197131537408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 _runtime‚6214.627070484 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386842212289024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚369082423670800 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381646008726016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114973572 _step‚370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56710018 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386842212289024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94663758 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209610984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114947228 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421182513342976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94663758 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5196221565952 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114947228 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5432187062784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84226402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198253010944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209682347 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209655975 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197343026176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426706103382528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56722083 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374340269836026 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114998429 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209682347 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426706347702784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114998429 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚381727609808896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386925880822784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381727627811840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56722081 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27504934 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚371 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94683918 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421272762987520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114972059 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94683918 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _runtime‚6215.903612144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209682347 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209655975 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5197343026176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426706103382528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220380160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56722083 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374340269836026 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114998429 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209682347 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426706347702784 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114998429 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚381727609808896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386925880822784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381727627811840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56722081 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27504934 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 _step‚371 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94683918 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421272762987520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114972059 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94683918 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 _runtime‚6215.903612144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84226402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198253010944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390979254005248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/aK3tšllocated_bytes.all.allocated‚390980181979648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373033320437360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5490801604608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385726963017216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5253218962432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211863886 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚385726945014272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5253218962432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85102695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27789427 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116192130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27788791 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚378266891950304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373033303398000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425693798565376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5491015706624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5232663537940 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5252308990976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211890232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5252308990976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116192130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57313268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425693822505472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431184600169984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚4599 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚390979254005248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95671756 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85102057ו¬Ûû'õO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5252308990976 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116192130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57313268 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425693822505472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431184600169984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 trainer/global_step‚4599 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚390979254005248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95671756 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85102057 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95671758 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95671758 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116218474 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚385726963017216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390979254005248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390980181979648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373033320437360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5490801604608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385726963017216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5253218962432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211863886 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚385726945014272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5253218962432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85102695 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27789427 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116192130 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27788791 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚378266891950304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373033303398000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425693798565376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5491015706624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5232663537940 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _step‚374 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5252308990976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211890232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ögu 2ž- _wandbruntime‚6275  _runtime‚6275@ ¤'ŸúM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116243331 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373112338031904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚6275.454996897 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5492192599040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385808582103040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚385808582103040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211935249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116243331 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116216961 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385808564100096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5234688786430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5254340435968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391062922539008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57325333 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373112320992544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431276240808960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85119911 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378347026818334 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚375 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5254340435968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95691918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27795219 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211935249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378346101803762 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431275988100096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378346101803762 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431275988100096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211935249 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116243331 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116216961 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385808564100096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5234688786430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5254340435968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252708864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228768768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚391062922539008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57325333 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373112320992544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431276240808960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85119911 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚378347026818334 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 _step‚375 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5254340435968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95691918 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27795219 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211935249÷D»7Ù2¡- _wandbruntime‚6275  _runtime‚6275,ɪgT¢Î 1loss/SRqqt1/assignment_loss‚2.293603458404541 1loss/SRqqt3/assignment_loss‚1.769373200237751 0loss/SRqqt4/detection_loss‚0.989699112580103 _runtime‚6275.457551857 0loss/FRt2/assignment_loss‚2.7109839536926965 /loss/FRt1/detection_loss‚2.1814609506425073 .loss/FRt3/detection_loss‚2.660457760721177 .loss/FRt4/detection_loss‚1.628649396439121 1loss/SRqqt3/detection_loss‚2.5096067905426027 /loss/FBt2/detection_loss‚0.9926588654737262 epoch‚3 /loss/FRt3/assignment_loss‚4.408750841111848 .loss/FBt1/detection_loss‚2.337365763882796 1loss/SRqqt4/assignment_loss‚1.030543073823973 /loss/FBt3/detection_loss‚1.9474662284438427 /loss/FBt4/detection_loss‚6.7451090812683105 1loss/FBt1/assignment_loss‚0.25888833528370014 0loss/FBt3/assignment_loss‚0.5135821439738231 0loss/SRqqt2/detection_loss‚2.345809531482783 trainer/global_step‚4599 0loss/FRt4/assignment_loss‚1.5941311741528446 0loss/FBt2/assignment_loss‚0.2022383530872805 /loss/FBt4/assignment_loss‚2.818598508834839 _step‚376 2loss/SRqqt2/assignment_loss‚1.8157038302584128 0loss/SRqqt1/detection_loss‚2.748871855735779 %loss/total_loss‚53.26331714317924 % _timestamp‚1.7885414801462028e+09 /loss/FRt1/assignment_loss‚4.013712053603315 /loss/FRt2/detection_loss‚2.7460528795014727E|MmY£Ó /loss/FRt1/assignment_loss‚4.013712053603315 /loss/FRt2/detection_loss‚2.7460528795014727 1loss/SRqqt1/assignment_loss‚2.293603458404541 1loss/SRqqt3/assignment_loss‚1.769373200237751 0loss/SRqqt4/detection_loss‚0.989699112580103 _runtime‚6275.457551857 0loss/FRt2/assignment_loss‚2.7109839536926965 /loss/FRt1/detection_loss‚2.1814609506425073 .loss/FRt3/detection_loss‚2.660457760721177 .loss/FRt4/detection_loss‚1.628649396439121 1loss/SRqqt3/detection_loss‚2.5096067905426027 /loss/FBt2/detection_loss‚0.9926588654737262 epoch‚3 /loss/FRt3/assignment_loss‚4.408750841111848 .loss/FBt1/detection_loss‚2.337365763882796 1loss/SRqqt4/assignment_loss‚1.030543073823973 /loss/FBt3/detection_loss‚1.9474662284438427 /loss/FBt4/detection_loss‚6.7451090812683105 1loss/FBt1/assignment_loss‚0.25888833528370014 0loss/FBt3/assignment_loss‚0.5135821439738231 0loss/SRqqt2/detection_loss‚2.345809531482783 trainer/global_step‚4599 0loss/FRt4/assignment_loss‚1.5941311741528446 0loss/FBt2/assignment_loss‚0.2022383530872805 /loss/FBt4/assignment_loss‚2.818598508834839 _step‚376 2loss/SRqqt2/assignment_loss‚1.8157038302584128 0loss/SRqqt1/detection_loss‚2.748871855735779 %loss/total_loss‚53.26331714317924 % _timestamp‚1.7885414801462028e+09ø,í]쓤‚(:ˆ ±ôëÔÍøÝË gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚175.651' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾3§V{¥‚(:ð ±ôëÔšŒÒÍ network.recv‚ 2.038856e+06 disk.sda1.in‚0 disk.dm-0.in‚15015.48828125 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚15015.48828125) disk.nvme0c0n1.out‚126158.89013671875' disk.nvme0n1.out‚126158.89013671875) disk./.usagePercent‚79.07598299520131$ disk./.usageGB‚1178.289291381836 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.out‚0.890625" disk.nvme0n1.in‚15015.48828125* proc.memory.percent‚18.634819984436035 disk.dm-1.out‚ 223.734375 disk.sda3.out‚ 223.734375" proc.memory.availableMB‚598.75# proc.memory.rssMB‚ 3053.12890625 proc.cpu.threads‚17 disk.sda.out‚ 223.734375 disk.sda1.out‚0$ disk.dm-0.out‚126158.89013671875 disk.sda2.out‚0 disk.sda3.in‚0$ memory_percent‚96.34552001953125 cpu‚25.35767009077398 network.sent‚ 1.1750317e+07®–˦‚(:ô ÀôëÔêÍ°Í disk.dm-1.out‚ 224.05859375 disk.sda3.in‚0 disk.sda.in‚0 disk.sda.out‚ 224.05859375# proc.memory.rssMB‚ 3053.12890625 cpu‚25.35714229763918 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚15015.48828125 disk.sda3.out‚ 224.05859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚15015.48828125 disk.dm-1.in‚0! proc.memory.availableMB‚598.5 network.sent‚ 1.1852924e+07 network.recv‚ 2.045262e+06$ disk./.usageGB‚1178.289234161377 disk.sda2.out‚0' disk.nvme0n1.out‚126504.57373046875$ disk.dm-0.out‚126504.57373046875# memory_percent‚96.3470458984375* proc.memory.percent‚18.634819984436035) disk./.usagePercent‚79.07597915508829 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.890625) disk.nvme0c0n1.out‚126504.57373046875" disk.nvme0n1.in‚15015.48828125_|«-’§‚(:‡ ÀôëԯÖΠgpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñŽV%“¨‚(:ˆ ÏôëÔ¿„äË gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚146.812' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.3624 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–¬žÉ†©‚(:û ÏôëÔ¹³³Ð disk.sda1.in‚0$ memory_percent‚96.34397029876709 cpu‚25.35655017692088$ disk.dm-0.out‚126802.28662109375 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 3053.12890625 proc.cpu.threads‚17 network.sent‚ 1.1861176e+07$ disk./.usageGB‚1178.289291381836 disk.sda2.out‚0 disk.drbd1260.out‚ 0.89453125" disk.nvme0n1.in‚15015.48828125 disk.sda.in‚0* proc.memory.percent‚18.634819984436035 network.recv‚ 2.050208e+06) disk./.usagePercent‚79.07598299520131 disk.dm-1.in‚0' disk.nvme0n1.out‚126802.28662109375 disk.sda.out‚ 225.1171875 disk.sda1.out‚0( proc.memory.availableMB‚ 599.00390625 disk.dm-0.in‚15015.48828125 disk.dm-1.out‚ 225.1171875$ disk.nvme0c0n1.in‚15015.48828125) disk.nvme0c0n1.out‚126802.28662109375 disk.sda3.out‚ 225.1171875Ž•ì…ª‚(:ú ÞôëÔäÅÀÐ disk.dm-1.out‚ 225.85546875 disk.sda2.out‚0* proc.memory.percent‚18.634819984436035 network.recv‚ 2.053596e+06) disk./.usagePercent‚79.07598094714103) disk.nvme0c0n1.out‚126865.06787109375 disk.sda3.in‚0 disk.sda3.out‚ 225.85546875 disk.sda1.out‚0 disk.dm-0.in‚15015.48828125 disk.sda2.in‚0 network.sent‚ 1.1866195e+07$ disk.nvme0c0n1.in‚15015.48828125$ disk.dm-0.out‚126865.06787109375 disk.dm-1.in‚0$ memory_percent‚96.34246826171875" proc.memory.availableMB‚599.25# proc.memory.rssMB‚ 3053.12890625 cpu‚25.356024658830556% disk./.usageGB‚1178.2892608642578 disk.drbd1260.in‚0' disk.nvme0n1.out‚126865.06787109375 disk.sda.out‚ 225.85546875 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.89453125" disk.nvme0n1.in‚15015.48828125 disk.sda.in‚0 disk.sda1.in‚0wƒ‚ú“«‚(:ˆ ÞôëÔœà„Ð gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž€Ç2¬- _wandbruntime‚6335  _runtime‚6335ìÔÅ­¿ & lr-AdamW/pg1‚0.0013028912633903534 & lr-AdamW/pg2‚0.0013028912633903534 trainer/global_step‚4649 # _timestamp‚1.78854153995801e+09 _runtime‚6335.269085217 _step‚377™a¤Ê®Ä & lr-AdamW/pg1‚0.0013028912633903534 & lr-AdamW/pg2‚0.0013028912633903534 trainer/global_step‚4649 # _timestamp‚1.78854153995801e+09 _runtime‚6335.269085217 _step‚377ùú÷2¯- _wandbruntime‚6335  _runtime‚6335·÷øÌö'°ðO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117463376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57916518 trainer/global_step‚4649 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389807917308416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5289448768052 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28079641 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117437032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885415399596887e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5288540793048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57916516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430205107727872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5308396416000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430205083787776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435754804113408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5549469700608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96679756 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚395116295721472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5549696385536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395116295721472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚382273648932612 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5309306387456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96679758 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389807917308416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85996157 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117437032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117463376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚6335.270978666 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389807899305472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96679756 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435754553488384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚382272723918248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚395117223695872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376984183125200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96679758 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376984200164560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214116788 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214143134 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚395116295721472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚382273648932612 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5309306387456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96679758 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389807917308416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85996157 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117437032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117463376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 _runtime‚6335.270978666 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389807899305472 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96679756 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435754553488384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚382272723918248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚395117223695872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376984183125200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96679758 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376984200164560 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214116788 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214143134 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 ]IDeviceStatsMonitor.on_train_batchøó¦€_start/active_bytes.large_pool.allocated‚389807917308416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5289448768052 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1195376640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28079641 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117437032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 % _timestamp‚1.7885415399596887e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5288540793048úuÃÎ]2²- _wandbruntime‚6336  _runtime‚6336!;¤ '³úM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117488233 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚395199964255232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435845950931456 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214161779 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57928581 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5309517876224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430295333432320 trainer/global_step‚4649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5290566041538 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117461863 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚379 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96699918 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚389889536394240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚395199964255232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389889536394240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5310427860992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389889518391296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430295309492224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96699918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117488233 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚382352858786070 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5550868110848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395199036267520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214188151 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28086082 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86014665 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5310427860992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86014024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57928583 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435846201543168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5550868110848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚395199036267520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214188151 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28086082 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86014665 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5310427860992 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86014024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57928583 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435846201543168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226671616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5309517876224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430295333432320 trainer/global_step‚4649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5290566041538 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117461863 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚379ûŽ¤G2µ- _wandbruntime‚6336  _runtime‚6336X‡É!T¶Î trainer/global_step‚4649 /loss/FRt2/assignment_loss‚2.470634761529091 /loss/FRt3/assignment_loss‚4.457285216876438 2loss/SRqqt2/assignment_loss‚1.4040350736384946 0loss/FBt3/assignment_loss‚0.6420960092155323 /loss/FBt1/detection_loss‚2.6245910111400814 1loss/SRqqt4/assignment_loss‚1.308822856021478 epoch‚3 $ _timestamp‚1.788541541635529e+09 _runtime‚6336.946402605 2loss/SRqqt1/assignment_loss‚1.8102193979116588 1loss/FBt2/assignment_loss‚0.16854783633154136 /loss/FRt4/detection_loss‚1.9596713908930432 /loss/FBt3/detection_loss‚1.1410896141567957 %loss/total_loss‚52.52665832997747 .loss/FRt4/assignment_loss‚2.13325579211397 1loss/SRqqt3/assignment_loss‚2.174391422134179 0loss/FBt4/assignment_loss‚2.7594497203826904 /loss/FRt2/detection_loss‚2.1201919244650083 1loss/SRqqt3/detection_loss‚2.7943676778903375 1loss/SRqqt4/detection_loss‚0.9987240480663984 0loss/FBt1/assignment_loss‚0.5686903386506148 .loss/FRt1/detection_loss‚2.335999685467458 1loss/SRqqt1/detection_loss‚2.6162880796652574 1loss/SRqqt2/detection_loss‚1.9452776802437646 _step‚380 /loss/FRt1/assignment_loss‚3.775668085790148 .loss/FRt3/detection_loss‚2.377825768987116 .loss/FBt2/detection_loss‚1.043417728018193 .loss/FBt4/detection_loss‚6.896117210388184’Lâ`Y·Ó 2loss/SRqqt1/assignment_loss‚1.8102193979116588 1loss/FBt2/assignment_loss‚0.16854783633154136 /loss/FRt4/detection_loss‚1.9596713908930432 /loss/FBt3/detection_loss‚1.1410896141567957 %loss/total_loss‚52.52665832997747 .loss/FRt4/assignment_loss‚2.13325579211397 1loss/SRqqt3/assignment_loss‚2.174391422134179 0loss/FBt4/assignment_loss‚2.7594497203826904 /loss/FRt2/detection_loss‚2.1201919244650083 1loss/SRqqt3/detection_loss‚2.7943676778903375 1loss/SRqqt4/detection_loss‚0.9987240480663984 0loss/FBt1/assignment_loss‚0.5686903386506148 .loss/FRt1/detection_loss‚2.335999685467458 1loss/SRqqt1/detection_loss‚2.6162880796652574 1loss/SRqqt2/detection_loss‚1.9452776802437646 _step‚380 /loss/FRt1/assignment_loss‚3.775668085790148 .loss/FRt3/detection_loss‚2.377825768987116 .loss/FBt2/detection_loss‚1.043417728018193 .loss/FBt4/detection_loss‚6.896117210388184 trainer/global_step‚4649 /loss/FRt2/assignment_loss‚2.470634761529091 /loss/FRt3/assignment_loss‚4.457285216876438 2loss/SRqqt2/assignment_loss‚1.4040350736384946 0loss/FBt3/assignment_loss‚0.6420960092155323 /loss/FBt1/detection_loss‚2.6245910111400814 1loss/SRqqt4/assignment_loss‚1.308822856021478 epoch‚3 $ _timestamp‚1.788541541635529e+09 _runtime‚6336.946402605üÜÊÁ®’¸‚(:‡ íôëÔŽÇùÌ gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù5逹‚(:õ íôëÔºÈóÏ disk.dm-0.in‚ 15154.2109375 network.sent‚ 1.1869447e+07 disk.drbd1260.out‚ 0.8984375! disk.nvme0n1.in‚ 15154.2109375$ disk.dm-0.out‚127090.67333984375 disk.dm-1.in‚0( proc.memory.availableMB‚ 598.96484375* proc.memory.percent‚18.634819984436035 cpu‚25.35554608636015 network.recv‚ 2.057135e+06) disk./.usagePercent‚79.07440470875272% disk./.usageGB‚1178.2657737731934# disk.nvme0c0n1.in‚ 15154.2109375 disk.sda3.in‚0 disk.sda2.out‚0' disk.nvme0n1.out‚127090.67333984375 disk.sda.out‚ 226.265625 disk.dm-1.out‚ 226.265625$ memory_percent‚96.34420871734619# proc.memory.rssMB‚ 3053.12890625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚127090.67333984375 disk.sda3.out‚ 226.265625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0¹çW“º‚(:ˆ üôëÔ°Œ‚Ì gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚116.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wõ‹z»‚(:ï üôëÔ³èÎ disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8984375 disk.sda3.in‚0 disk.sda.out‚ 226.59375 disk.dm-0.in‚ 15154.2109375 disk.dm-1.in‚0 cpu‚25.354995956426443$ disk./.usageGB‚1178.289291381836 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 15154.2109375) disk.nvme0c0n1.out‚127489.88818359375 disk.sda.in‚0$ disk.dm-0.out‚127489.88818359375* proc.memory.percent‚18.634819984436035 network.recv‚ 2.063803e+06! disk.nvme0n1.in‚ 15154.2109375' disk.nvme0n1.out‚127489.88818359375 disk.sda3.out‚ 226.59375 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.1978078e+07) disk./.usagePercent‚79.07598299520131 disk.dm-1.out‚ 226.59375$ memory_percent‚96.34418487548828% proc.memory.availableMB‚ 598.96875# proc.memory.rssMB‚ 3053.12890625 proc.cpu.threads‚17WF¥„“¼‚(:ˆ ‹õëÔ¹í÷Ë gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E¦†½‚(:û ‹õëÔÆ¢öÎ) disk./.usagePercent‚79.05885839522846 disk.dm-1.out‚ 226.95703125) disk.nvme0c0n1.out‚127814.24072265625 disk.sda3.in‚0 disk.sda.out‚ 226.95703125 cpu‚25.35451649195225$ disk.dm-0.out‚127814.24072265625 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 15154.2109375 disk.sda3.out‚ 226.95703125 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚ 2.068519e+06$ disk./.usageGB‚1178.034122467041 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 15154.2109375 disk.sda1.out‚0$ memory_percent‚96.34416103363037( proc.memory.availableMB‚ 598.97265625# proc.memory.rssMB‚ 3053.12890625 disk.dm-0.in‚ 15154.2109375 disk.drbd1260.out‚ 0.90234375' disk.nvme0n1.out‚127814.24072265625* proc.memory.percent‚18.634819984436035 proc.cpu.threads‚17 network.sent‚ 1.1984166e+07Í€š“¾‚(:ˆ šõëÔÄóÍ gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å»·m„¿‚(:ù šõëÔíçÑ! disk.nvme0n1.in‚ 15154.2109375' disk.nvme0n1.out‚128159.92041015625# proc.memory.rssMB‚ 3053.12890625 network.sent‚ 1.1989239e+07 disk.dm-1.in‚0 disk.dm-1.out‚ 227.42578125 disk.sda3.in‚0* proc.memory.percent‚18.634819984436035% disk./.usageGB‚1178.0313911437988 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.90234375 disk.sda3.out‚ 227.42578125 disk.sda1.out‚0$ memory_percent‚96.34571075439453% proc.memory.availableMB‚ 598.71875 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 15154.2109375 disk.sda.in‚0 disk.sda.out‚ 227.42578125 disk.sda1.in‚0 cpu‚25.35401700988623 proc.cpu.threads‚17) disk.nvme0c0n1.out‚128159.92041015625 network.recv‚ 2.072177e+06) disk./.usagePercent‚79.05867509383393 disk.dm-0.in‚ 15154.2109375$ disk.dm-0.out‚128159.92041015625²ikÌ2À- _wandbruntime‚6397  _runtime‚6397\0K§ÇÁÁ & lr-AdamW/pg1‚0.0012926512239722254 & lr-AdamW/pg2‚0.0012926512239722254 trainer/global_step‚4699 % _timestamp‚1.7885416018862734e+09 _runtime‚6397.197271671 _step‚381õÌÂÆ trainer/global_step‚4699 % _timestamp‚1.7885416018862734e+09 _runtime‚6397.197271671 _step‚381 & lr-AdamW/pg1‚0.0012926512239722254 & lr-AdamW/pg2‚0.0012926512239722254ýÆò†2Ã- _wandbruntime‚6397  _runtime‚6397¾ >˜ ÄðO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5608235524096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216369690 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399254265412096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5364483841024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393888871599616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86889517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58519766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28369136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434716369010176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399253337437696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5608449626112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97687758 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 M@DeviceStatsMonitor.oîæ±A^n_train_batch_start/inactive_split.all.freed‚86888902 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393888871599616 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _runtime‚6397.198489466 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118708278 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380935079891760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118708278 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118681934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393888853596672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380935062852400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393888853596672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5364483841024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58519768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5365393812480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5344418048156 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434716392950272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5365393812480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97687758 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97687756 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399253337437696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97687756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440324604534272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0jC˜û'ÅõO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380935079891760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118708278 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238042112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118681934 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚393888853596672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214102016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380935062852400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚393888853596672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5364483841024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58519768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5365393812480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5344418048156 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434716392950272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5365393812480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97687758 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97687756 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399253337437696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97687756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440324604534272 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5608235524096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216369690 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399254265412096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5364483841024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚393888871599616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86889517 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58519766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28369136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434716369010176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399253337437696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5608449626112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97687758 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86888902 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚393888871599616 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 _runtime‚6397.198489466 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118708278 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400þêêyµ2Æ- _wandbruntime‚6398  _runtime‚6398kã\'ÇúM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118733135 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚393970490685440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399337005971456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440416247488512 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386359615768378 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86906730 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434806594714624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚393970490685440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97707916 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216441053 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86907350 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28374899 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399337005971456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚4699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5366515286016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885416031369567e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216441053 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28375517 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386360540782950 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚6398.448571078 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5365605301248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381014097486304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216414681 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5609628833792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5346443296646 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5366515286016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118706765 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434806618654720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97707918 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118706765 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚393970472682496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚383 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97707916 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚383 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97707916 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118733135 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚393970490685440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚399337005971456 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440416247488512 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚386359615768378 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86906730 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434806594714624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13205766144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚393970490685440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97707916 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216441053 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 909984768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86907350 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28374899 RBDeviceStatsMonitor.on_train_batch_end/reserved_byteÜlEQ³s.all.allocated‚ 14401142784 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚399337005971456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 trainer/global_step‚4699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5366515286016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885416031369567e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216441053 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28375517 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚386360540782950 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚570 _runtime‚6398.448571078 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5365605301248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381014097486304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216414681 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5609628833792 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5346443296646 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5366515286016 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118706765 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434806618654720ÿFô2É- _wandbruntime‚6398  _runtime‚6398³:*½Dʾ 1loss/SRqqt1/detection_loss‚3.9882578362118113 1loss/SRqqt3/detection_loss‚2.7291366308927536 0loss/FRt1/assignment_loss‚3.0561083832452463 0loss/FRt2/assignment_loss‚2.5378031063079836 2loss/SRqqt4/assignment_loss‚1.0590990171728967 .loss/FBt2/detection_loss‚1.313349555082181 trainer/global_step‚4699 $ _timestamp‚1.788541603139893e+09 1loss/SRqqt2/assignment_loss‚1.555054943888418 0loss/FBt3/assignment_loss‚0.6293140743748217 .loss/FRt1/detection_loss‚2.241436782617902 -loss/FRt2/detection_loss‚2.81110187292099 0loss/SRqqt2/detection_loss‚1.940594978870884 1loss/SRqqt4/detection_loss‚1.0656221434473991 1loss/FBt2/assignment_loss‚0.12575233341709635 loss/FBt4/assignment_loss‚0 -loss/FBt1/detection_loss‚2.89789039293925 /loss/FBt3/detection_loss‚1.2786826650644172 /loss/FBt4/detection_loss‚0.8763856887817383 /loss/FRt4/detection_loss‚1.8729627506186566 epoch‚3 0loss/FRt4/assignment_loss‚1.6960589081048965 &loss/total_loss‚44.506629125334655 /loss/FRt3/assignment_loss‚3.780696944395701 .loss/FRt3/detection_loss‚2.650566909178936 _runtime‚6398.450676926 _step‚384 2loss/SRqqt1/assignment_loss‚1.7261236093261025 1loss/SRqqt3/assignment_loss‚2.162886897900275 0loss/FBt1/assignment_loss‚0.5117427005742987µ µxIËà .loss/FRt1/detection_loss‚2.241436782617902 -loss/FRt2/detection_loss‚2.81110187292099 0loss/SRqqt2/detection_loss‚1.940594978870884 1loss/SRqqt4/detection_loss‚1.0656221434473991 1loss/FBt2/assignment_loss‚0.12575233341709635 loss/FBt4/assignment_loss‚0 -loss/FBt1/detection_loss‚2.89789039293925 /loss/FBt3/detection_loss‚1.2786826650644172 /loss/FBt4/detection_loss‚0.8763856887817383 /loss/FRt4/detection_loss‚1.8729627506186566 epoch‚3 0loss/FRt4/assignment_loss‚1.6960589081048965 &loss/total_loss‚44.506629125334655 /loss/FRt3/assignment_loss‚3.780696944395701 .loss/FRt3/detection_loss‚2.650566909178936 _runtime‚6398.450676926 _step‚384 2loss/SRqqt1/assignment_loss‚1.7261236093261025 1loss/SRqqt3/assignment_loss‚2.162886897900275 0loss/FBt1/assignment_loss‚0.5117427005742987 1loss/SRqqt1/detection_loss‚3.9882578362118113 1loss/SRqqt3/detection_loss‚2.7291366308927536 0loss/FRt1/assignment_loss‚3.0561083832452463 0loss/FRt2/assignment_loss‚2.5378031063079836 2loss/SRqqt4/assignment_loss‚1.0590990171728967 .loss/FBt2/detection_loss‚1.313349555082181 trainer/global_step‚4699 $ _timestamp‚1.788541603139893e+09 1loss/SRqqt2/assignment_loss‚1.555054943888418 0loss/FBt3/assignment_loss‚0.6293140743748217€‹¸|¤“Ì‚(:ˆ ©õëÔ¦×æÊ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢‘€Í‚(:õ ©õëÔ¬”´Í network.recv‚ 2.075688e+06$ disk.nvme0c0n1.in‚15324.19921875% disk.nvme0n1.out‚128228.740234375 disk.sda3.in‚0 disk.sda3.out‚ 228.07421875 disk.dm-1.out‚ 228.07421875 disk.sda2.in‚0 disk.drbd1260.in‚0% disk./.usageGB‚1178.0362930297852" disk.nvme0n1.in‚15324.19921875 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚15324.19921875" disk.dm-0.out‚128228.740234375( proc.memory.availableMB‚ 598.65234375# proc.memory.rssMB‚ 3053.12890625 network.sent‚ 1.2007531e+07' disk.nvme0c0n1.out‚128228.740234375 disk.sda1.out‚0" memory_percent‚96.346116065979* proc.memory.percent‚18.634819984436035 cpu‚25.353461767812306 proc.cpu.threads‚17) disk./.usagePercent‚79.05900406351543 disk.sda.out‚ 228.07421875 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚0.90625vŸ(“΂(:ˆ ¸õëÔªœÌ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9962 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¨Ï‚(:ô ¸õëÔ½Ï¾Ï network.sent‚ 1.2101991e+07% disk./.usageGB‚1178.0127906799316 disk.sda2.out‚0' disk.nvme0c0n1.out‚128430.224609375% disk.nvme0n1.out‚128430.224609375 disk.sda.in‚0 disk.dm-0.in‚ 15353.0546875$ memory_percent‚96.34311199188232 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda.out‚ 228.40234375 disk.sda1.in‚0 disk.dm-1.out‚ 228.40234375( proc.memory.availableMB‚ 599.14453125 disk.sda3.out‚ 228.40234375 disk.drbd1260.out‚0.90625# disk.nvme0c0n1.in‚ 15353.0546875! disk.nvme0n1.in‚ 15353.0546875" disk.dm-0.out‚128430.224609375# proc.memory.rssMB‚ 3053.12890625* proc.memory.percent‚18.634819984436035 cpu‚25.352934740825372 network.recv‚ 2.081089e+06) disk./.usagePercent‚79.05742680109698 disk.sda1.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17Úþú/“Ђ(:ˆ ÇõëÔ‡ùØÌ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚154.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ší.<~Ñ‚(:ó ÇõëÔ°€¾Ð disk.sda1.out‚0 disk.dm-0.in‚ 15353.0546875 disk.dm-1.in‚0 disk.dm-1.out‚ 229.03125( proc.memory.availableMB‚ 598.37890625 network.recv‚ 2.085623e+06) disk./.usagePercent‚79.07236816881785% disk./.usageGB‚1178.2354278564453 disk.sda2.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 15353.0546875 disk.sda.in‚0$ memory_percent‚96.34778499603271 network.sent‚ 1.2108229e+07# disk.nvme0c0n1.in‚ 15353.0546875 disk.sda.out‚ 229.03125* proc.memory.percent‚18.634819984436035 proc.cpu.threads‚17 disk.sda3.out‚ 229.03125 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0$ disk.dm-0.out‚128909.53076171875# proc.memory.rssMB‚ 3053.12890625 cpu‚25.35249615568209 disk.drbd1260.out‚ 0.91015625) disk.nvme0c0n1.out‚128909.53076171875' disk.nvme0n1.out‚128909.53076171875Jr"f“Ò‚(:ˆ ÖõëÔßßöË gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Õ§†Ó‚(:û ÖõëÔ½µ½Ï cpu‚25.352113137646764' disk.nvme0n1.out‚129320.79638671875 network.sent‚ 1.2113265e+07 network.recv‚ 2.089029e+06( disk./.usagePercent‚79.0842709831148% disk./.usageGB‚1178.4127883911133 disk.drbd1260.in‚0 disk.sda.out‚ 229.47265625 disk.sda1.out‚0! disk.nvme0n1.in‚ 15353.0546875 disk.sda1.in‚0$ disk.dm-0.out‚129320.79638671875 disk.dm-1.out‚ 229.47265625# proc.memory.rssMB‚ 3053.12890625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 229.47265625 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 15353.0546875 disk.sda.in‚0 disk.dm-0.in‚ 15353.0546875 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.91015625) disk.nvme0c0n1.out‚129320.79638671875$ memory_percent‚96.34623527526855' proc.memory.availableMB‚ 598.6328125* proc.memory.percent‚18.634819984436035ô/2Ô- _wandbruntime‚6458  _runtime‚6458eo]¯ÇÕÁ & lr-AdamW/pg2‚0.0012823174250522036 trainer/global_step‚4749 % _timestamp‚1.7885416633434417e+09 _runtime‚6458.654638998 _step‚385 & lr-AdamW/pg1‚0.0012823174250522036ç‡8ÌÖÆ & lr-AdamW/pg1‚0.0012823174250522036 & lr-AdamW/pg2‚0.0012823174250522036 trainer/global_step‚4749 % _timestamp‚1.7885416633434417e+09 _runtime‚6458.654638998 _step‚385€s¶Y2×- _wandbruntime‚6458  _runtime‚6458ËÙÛ*ö'ØðO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218622592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28660687 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444894393912832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98695758 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98695756 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119926836 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚444894638246400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384885959618960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5401203278268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397969807887872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119953180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403391307128320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384885942579600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397969825890816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 _runtime‚6458.656676789 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87783705 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87783077 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98695756 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403390379153920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5420571266048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439227678172672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666739680256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390287162897228 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666960073728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚4749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5420571266048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119953180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28660061 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403390379153920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268o³?þÔ"ÙõO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98695756 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403390379153920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5420571266048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439227678172672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666739680256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390287162897228 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666960073728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220393472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 trainer/global_step‚4749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5420571266048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119953180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28660061 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403390379153920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218622592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28660687 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444894393912832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98695758 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98695756 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119926836 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚444894638246400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384885959618960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5401203278268 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397969807887872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_poolØÕ÷Ó'.peak‚44 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119953180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403391307128320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384885942579600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚397969825890816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 _runtime‚6458.656676789 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87783705 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87783077 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0‚¢Éä2Ú- _wandbruntime‚6460  _runtime‚6460xh¿ý'ÛúM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403473119699968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403474047687680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59135081 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439317879937024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398051426973696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚4749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5422602711040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403474047687680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218693955 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59135083 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5402320551754 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98715916 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119951667 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚387 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398051444976640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119978037 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390366372750686 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218667583 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119951667 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98715918 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28666577 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384964977213504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384964960174144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5421692726272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87801011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390367297765258 _runtime‚6460.264576834 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218693955m÷mÍ'ÜÿM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 14135738368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403473119699968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403474047687680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59135081 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚439317879937024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398051426973696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1195376640 trainer/global_step‚4749 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 13002869760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5422602711040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403474047687680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 925014572 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218693955 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59135083 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5402320551754 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 909984768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98715916 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119951667 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚387 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398051444976640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119978037 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390366372750686 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12553587712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1185 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218667583 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119951667 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98715918 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28666577 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384964977213504 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384964960174144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚570 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5421692726272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87801011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390367297765258 _runtime‚6460.264576834 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚570 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218693955ƒYÉ£þ2Ý- _wandbruntime‚6460  _runtime‚6460ÊüyFÞÀ /loss/FRt2/assignment_loss‚2.049916847394063 /loss/FRt3/detection_loss‚3.1440072555734777 /loss/FBt4/detection_loss‚0.5495762825012207 epoch‚3 trainer/global_step‚4749 0loss/SRqqt3/detection_loss‚2.867617677558552 % _timestamp‚1.7885416649563487e+09 2loss/SRqqt3/assignment_loss‚2.1141868152401666 /loss/FRt1/detection_loss‚2.6617953355113664 .loss/FRt4/detection_loss‚1.808488714892198 $loss/total_loss‚43.9680496967037 0loss/FRt4/assignment_loss‚1.6084255023914225 1loss/SRqqt4/detection_loss‚0.9806244101262453 _step‚388 /loss/FRt1/assignment_loss‚3.408581489171737 /loss/FBt2/detection_loss‚0.9764985704826097 _runtime‚6460.267433949 1loss/SRqqt1/assignment_loss‚2.286590255075885 1loss/SRqqt4/assignment_loss‚1.186393272272791 0loss/FBt3/assignment_loss‚0.5253787535155103 1loss/SRqqt1/detection_loss‚2.5241945722410755 1loss/SRqqt2/detection_loss‚2.2499659369389216 .loss/FBt3/detection_loss‚2.431291851927252 /loss/FRt3/assignment_loss‚3.367372995331174 2loss/SRqqt2/assignment_loss‚1.7650323717544476 1loss/FBt2/assignment_loss‚0.12623546547410663 loss/FBt4/assignment_loss‚0 1loss/FBt1/assignment_loss‚0.37578571096528324 .loss/FRt2/detection_loss‚2.600462919817521 /loss/FBt1/detection_loss‚2.3596266905466714½vW´KßÅ /loss/FRt2/assignment_loss‚2.049916847394063 /loss/FRt3/detection_loss‚3.1440072555734777 /loss/FBt4/detection_loss‚0.5495762825012207 epoch‚3 trainer/global_step‚4749 0loss/SRqqt3/detection_loss‚2.867617677558552 % _timestamp‚1.7885416649563487e+09 2loss/SRqqt3/assignment_loss‚2.1141868152401666 /loss/FRt1/detection_loss‚2.6617953355113664 .loss/FRt4/detection_loss‚1.808488714892198 $loss/total_loss‚43.9680496967037 0loss/FRt4/assignment_loss‚1.6084255023914225 1loss/SRqqt4/detection_loss‚0.9806244101262453 _step‚388 /loss/FRt1/assignment_loss‚3.408581489171737 /loss/FBt2/detection_loss‚0.9764985704826097 _runtime‚6460.267433949 1loss/SRqqt1/assignment_loss‚2.286590255075885 1loss/SRqqt4/assignment_loss‚1.186393272272791 0loss/FBt3/assignment_loss‚0.5253787535155103 1loss/SRqqt1/detection_loss‚2.5241945722410755 1loss/SRqqt2/detection_loss‚2.2499659369389216 .loss/FBt3/detection_loss‚2.431291851927252 /loss/FRt3/assignment_loss‚3.367372995331174 2loss/SRqqt2/assignment_loss‚1.7650323717544476 1loss/FBt2/assignment_loss‚0.12623546547410663 loss/FBt4/assignment_loss‚0 1loss/FBt1/assignment_loss‚0.37578571096528324 .loss/FRt2/detection_loss‚2.600462919817521 /loss/FBt1/detection_loss‚2.3596266905466714„˜ÄÜà‚(:’ åõëÔŒ©Ï gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚32.0 gpu.0.powerWatts‚96.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.264799999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûæ×”á‚(:ô åõëÔò•åÐ network.recv‚ 2.092756e+06 disk.sda2.out‚0! disk.nvme0n1.in‚ 15353.0546875 disk.sda3.out‚ 229.8984375 disk.sda1.in‚0 disk.dm-0.in‚ 15353.0546875$ disk.dm-0.out‚129883.42724609375 disk.dm-1.in‚0 network.sent‚ 1.2118408e+07) disk.nvme0c0n1.out‚129883.42724609375 disk.sda.out‚ 229.8984375% proc.memory.availableMB‚ 598.59375* proc.memory.percent‚18.634843826293945 cpu‚25.351675286946136 disk.drbd1260.out‚ 0.9140625# disk.nvme0c0n1.in‚ 15353.0546875' disk.nvme0n1.out‚129883.42724609375 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚96.34647369384766 proc.cpu.threads‚17) disk./.usagePercent‚79.08427738330316$ disk./.usageGB‚1178.412883758545 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 229.8984375" proc.memory.rssMB‚ 3053.1328125IÈ¡žâ‚(:“ ôõëÔµÿÉÌ gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚165.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.062200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^iBìxã‚(:í ôõëÔӾΠnetwork.sent‚ 1.2226832e+07$ disk./.usageGB‚1178.436294555664 disk.sda.out‚ 230.453125 disk.dm-1.out‚ 230.453125$ memory_percent‚96.34647369384766% proc.memory.availableMB‚ 598.59375" proc.memory.rssMB‚ 3053.1328125* proc.memory.percent‚18.634843826293945 network.recv‚ 2.099268e+06( disk./.usagePercent‚79.0858485015408 disk.drbd1260.out‚ 0.9140625' disk.nvme0n1.out‚130042.29443359375 disk.sda3.out‚ 230.453125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 15559.609375 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 15559.609375 disk.sda3.in‚0 disk.sda1.out‚0$ disk.dm-0.out‚130042.29443359375 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚130042.29443359375 disk.nvme0n1.in‚ 15559.609375 cpu‚25.351154121136577 proc.cpu.threads‚17xc9ˆžä‚(:“ ƒöëÔúª÷Ë gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.586399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‹zczå‚(:ï ƒöëÔû­àÏ disk.sda2.out‚0& disk.nvme0n1.out‚130133.5830078125 disk.sda3.out‚ 231.1953125 network.sent‚ 1.223308e+07 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 15559.609375 disk.sda.in‚0$ memory_percent‚96.34644985198975* proc.memory.percent‚18.634843826293945$ disk./.usageGB‚1178.436351776123# disk.dm-0.out‚130133.5830078125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.91796875" disk.nvme0c0n1.in‚ 15559.609375( disk.nvme0c0n1.out‚130133.5830078125 disk.sda3.in‚0( proc.memory.availableMB‚ 598.59765625 disk.sda2.in‚0 disk.sda.out‚ 231.1953125" proc.memory.rssMB‚ 3053.1328125 cpu‚25.35068284728139 proc.cpu.threads‚17 network.recv‚ 2.103798e+06( disk./.usagePercent‚79.0858523416538 disk.sda1.in‚0 disk.dm-0.in‚ 15559.609375 disk.dm-1.out‚ 231.19531259²¸Z“æ‚(:ˆ ’öëÔ¶½úÎ gpu.0.gpu‚32.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚117.539' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ÙVQç‚(:ö ’öëÔõòÔ network.sent‚ 1.2238264e+07% disk./.usageGB‚1178.4363708496094( disk.nvme0c0n1.out‚130292.5244140625 disk.sda.in‚0( proc.memory.availableMB‚ 598.59765625* proc.memory.percent‚18.634843826293945 cpu‚25.350113369263788) disk./.usagePercent‚79.08585362169148 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.91796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 231.80859375$ memory_percent‚96.34644985198975 network.recv‚ 2.107521e+06 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 15559.609375& disk.nvme0n1.out‚130292.5244140625 disk.sda3.out‚ 231.80859375 disk.sda.out‚ 231.80859375 disk.dm-0.in‚ 15559.609375# disk.dm-0.out‚130292.5244140625 disk.nvme0n1.in‚ 15559.609375 disk.sda3.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 3053.1328125 proc.cpu.threads‚17;áf2è- _wandbruntime‚6523  _runtime‚6523iRjÅé¿ % lr-AdamW/pg2‚0.001271892020178494 trainer/global_step‚4799 % _timestamp‚1.7885417282872593e+09 _runtime‚6523.598258873 _step‚389 % lr-AdamW/pg1‚0.001271892020178494ãq»ÊêÄ % lr-AdamW/pg1‚0.001271892020178494 % lr-AdamW/pg2‚0.001271892020178494 trainer/global_step‚4799 % _timestamp‚1.7885417282872593e+09 _runtime‚6523.598258873 _step‚389…O–ÿO2ë- _wandbruntime‚6523  _runtime‚6523í…¿ ìðO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99703756 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407527420870144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443738963395072 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407527420870144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121171738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28950474 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5725267130880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121171738 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121198082 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885417282888856e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394292994865172 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388836839346160 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.largIžÊˆôe_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5477568662528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220901840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5725493815808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449464457210880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121198082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402050780182016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5477568662528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99703758 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88677363 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407528348844544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28951095 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚6523.600246813 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449464206585856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚402050762179072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99703756 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59726268 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88676740 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394293919879536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99703758 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚402050762179072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5476658691072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443738939454976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388836822306800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5456172558372 _step‚390 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220875494 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5476658691072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407528348844544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402050780182016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5457080533376z¼û'íõO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 409073152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394293919879536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99703758 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚402050762179072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5476658691072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443738939454976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388836822306800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5456172558372 _step‚390 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220875494 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5476658691072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407528348844544 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚402050780182016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5457080533376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99703756 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407527420870144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1231 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443738963395072 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407527420870144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121171738 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28950474 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5725267130880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121171738 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121198082 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885417282888856e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394292994865172 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388836839346160 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5477568662528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220901840 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5725493815808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449464457210880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121198082 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚402050780182016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5477568662528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99703758 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88677363 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407528348844544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28951095 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚6523.600246813 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449464206585856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚402050762179072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99703756 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59726268 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88676740 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0†ã8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407611089403904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220946857 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121196569 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88694590 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449555851931136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99723916 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121222939 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99723918 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5477780151296 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449555597125120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443829165159424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5477780151296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚402132381264896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407610161416192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220920485 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5726431965696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394373129732994 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4799 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407611089403904 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5478690136064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388915839901344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99723918 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121222939 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402132381264896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394374054747566 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220946857 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚402132381264896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 907975212 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 927987712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407610161416192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220920485 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚44 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5726431965696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3725 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394373129732994 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1185 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4799 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407611089403904 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5478690136064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 13002869760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388915839901344 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99723918 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121222939 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚402132381264896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 409073152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394374054747566 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ?6De˜·£LGviceStatsMonitor.on_train_batch_end/num_device_alloc‚1185 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220946857 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14401142784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407611089403904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220946857 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚615 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121196569 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88694590 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449555851931136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 353397248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1187728393 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14401142784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99723916 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121222939 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚615 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99723918 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13684416521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0‡{LFÌ2ñ- _wandbruntime‚6524  _runtime‚6524¼ hZòÔ 1loss/SRqqt2/assignment_loss‚2.072441943530594 .loss/FRt2/detection_loss‚2.561629641997187 /loss/FBt3/detection_loss‚1.5588071147212759 %loss/total_loss‚51.86196844095544 /loss/FRt3/assignment_loss‚3.450587589825903 2loss/SRqqt3/assignment_loss‚1.8882876519233949 1loss/FBt2/assignment_loss‚0.22426137691583237 /loss/FRt4/detection_loss‚1.8261461723595858 1loss/SRqqt2/detection_loss‚1.7284540058143678 epoch‚3 trainer/global_step‚4799 0loss/FRt2/assignment_loss‚2.6718016360935413 1loss/SRqqt4/assignment_loss‚1.403299393764298 1loss/FBt1/assignment_loss‚0.48713017377927564 /loss/FRt3/detection_loss‚2.8486791680763375 1loss/SRqqt1/detection_loss‚2.3085784630195514 1loss/SRqqt4/detection_loss‚1.0948033651470073 /loss/FBt1/detection_loss‚2.0911612255232677 _runtime‚6524.755902267 -loss/FRt1/detection_loss‚2.31557809561491 /loss/FBt4/detection_loss‚7.0572943687438965 0loss/FBt3/assignment_loss‚0.7355377673442547 1loss/SRqqt3/detection_loss‚2.5685112918576887 /loss/FBt2/detection_loss‚0.7893854845792819 0loss/FBt4/assignment_loss‚2.7836971282958984 _step‚392 2loss/SRqqt1/assignment_loss‚1.9874223913695361 $ _timestamp‚1.788541729444384e+09 0loss/FRt1/assignment_loss‚3.5305196987258065 0loss/FRt4/assignment_loss‚1.8779532919327417Ž¾_óÙ 0loss/FRt1/assignment_loss‚3.5305196987258065 0loss/FRt4/assignment_loss‚1.8779532919327417 1loss/SRqqt2/assignment_loss‚2.072441943530594 .loss/FRt2/detection_loss‚2.561629641997187 /loss/FBt3/detection_loss‚1.5588071147212759 %loss/total_loss‚51.86196844095544 /loss/FRt3/assignment_loss‚3.450587589825903 2loss/SRqqt3/assignment_loss‚1.8882876519233949 1loss/FBt2/assignment_loss‚0.22426137691583237 /loss/FRt4/detection_loss‚1.8261461723595858 1loss/SRqqt2/detection_loss‚1.7284540058143678 epoch‚3 trainer/global_step‚4799 0loss/FRt2/assignment_loss‚2.6718016360935413 1loss/SRqqt4/assignment_loss‚1.403299393764298 1loss/FBt1/assignment_loss‚0.48713017377927564 /loss/FRt3/detection_loss‚2.8486791680763375 1loss/SRqqt1/detection_loss‚2.3085784630195514 1loss/SRqqt4/detection_loss‚1.0948033651470073 /loss/FBt1/detection_loss‚2.0911612255232677 _runtime‚6524.755902267 -loss/FRt1/detection_loss‚2.31557809561491 /loss/FBt4/detection_loss‚7.0572943687438965 0loss/FBt3/assignment_loss‚0.7355377673442547 1loss/SRqqt3/detection_loss‚2.5685112918576887 /loss/FBt2/detection_loss‚0.7893854845792819 0loss/FBt4/assignment_loss‚2.7836971282958984 _step‚392 2loss/SRqqt1/assignment_loss‚1.9874223913695361 $ _timestamp‚1.788541729444384e+09ˆ_Qˆý“ô‚(:ˆ ¡öëÔîáàË gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÿ¢ç}õ‚(:ò ¡öëÔªŸ˜Í$ memory_percent‚96.34823799133301% disk./.usageGB‚1178.2720756530762 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚0.921875 disk.sda1.out‚0) disk./.usagePercent‚79.07482763319929 disk.sda2.in‚0 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚130542.1220703125 disk.sda.out‚ 232.22265625' proc.memory.availableMB‚ 598.3046875" proc.memory.rssMB‚ 3053.1328125* proc.memory.percent‚18.634843826293945 network.recv‚ 2.111281e+06 disk.dm-0.in‚ 15559.609375# disk.dm-0.out‚130542.1220703125 disk.dm-1.out‚ 232.22265625& disk.nvme0n1.out‚130542.1220703125 disk.sda3.in‚0 cpu‚25.34967333706203 proc.cpu.threads‚17 network.sent‚ 1.2243473e+07" disk.nvme0c0n1.in‚ 15559.609375 disk.nvme0n1.in‚ 15559.609375 disk.sda3.out‚ 232.22265625 disk.sda.in‚0 disk.sda1.in‚01™÷0žö‚(:“ °öëÔ¢Ü¶Ì gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.944' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.388799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ÁiŽs÷‚(:è °öëÔÛ¥øÎ( disk.nvme0c0n1.out‚130797.3330078125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.34537696838379 network.sent‚ 1.2354019e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda.out‚232.8125# disk.dm-0.out‚130797.3330078125 disk.dm-1.out‚232.8125) disk./.usagePercent‚79.04240351096516$ disk./.usageGB‚1177.788932800293& disk.nvme0n1.out‚130797.3330078125 disk.dm-1.in‚0' proc.memory.availableMB‚ 598.7734375 cpu‚25.34920445971536 disk.sda3.out‚232.8125 network.recv‚ 2.117162e+06 disk.drbd1260.out‚0.921875! disk.nvme0n1.in‚ 15721.8046875 disk.dm-0.in‚ 15721.8046875" proc.memory.rssMB‚ 3053.1328125* proc.memory.percent‚18.634843826293945 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 15721.8046875('“4“ø‚(:ˆ ¿öëÔÑüÇÌ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.962' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾¿ÁFù‚(:ô ¿öëÔÝ”Ñ disk.sda2.out‚0 disk.drbd1260.out‚ 0.92578125 disk.sda3.out‚ 233.484375 disk.sda1.in‚0 disk.dm-1.out‚ 233.484375$ memory_percent‚96.34687900543213( proc.memory.availableMB‚ 598.52734375 network.recv‚ 2.12184e+06% disk./.usageGB‚1177.8005867004395# disk.nvme0c0n1.in‚ 15721.8046875! disk.nvme0n1.in‚ 15721.8046875" proc.memory.rssMB‚ 3053.1328125* proc.memory.percent‚18.634843826293945 proc.cpu.threads‚17 network.sent‚ 1.2358341e+07' disk.nvme0n1.out‚131161.83349609375 disk.sda.in‚0 disk.sda.out‚ 233.484375 disk.dm-0.in‚ 15721.8046875 cpu‚25.34865843344119 disk.sda2.in‚0 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚131161.83349609375 disk.sda3.in‚0 disk.sda1.out‚0$ disk.dm-0.out‚131161.83349609375 disk.dm-1.in‚0) disk./.usagePercent‚79.04318561398226]¡¸.’ú‚(:‡ ÎöëÔþíÞÌ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼Ç=û‚(:ô ÎöëÔ…óýÏ! disk.nvme0n1.in‚ 15721.8046875 disk.sda3.in‚0* proc.memory.percent‚18.634843826293945 cpu‚25.35052189057486 network.recv‚ 2.125357e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚131265.18896484375 disk.sda1.out‚0" proc.memory.rssMB‚ 3053.1328125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1260.out‚ 0.92578125 disk.sda1.in‚0$ memory_percent‚96.34537696838379' proc.memory.availableMB‚ 598.7734375) disk./.usagePercent‚79.04318202987677 disk.dm-0.in‚ 15721.8046875$ disk.dm-0.out‚131265.18896484375 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 15721.8046875 disk.sda3.out‚ 234.171875 disk.sda.in‚0 disk.sda.out‚ 234.171875 network.sent‚ 1.2365291e+07% disk./.usageGB‚1177.8005332946777 disk.dm-1.out‚ 234.171875) disk.nvme0c0n1.out‚131265.18896484375L¨ü‚(:’ ÝöëÔ»•ŠÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.507' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.701400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g42 ý‚(:ô ÝöëÔíÆ´Î network.recv‚ 2.128919e+06) disk./.usagePercent‚79.04239096659597" disk.dm-0.out‚131458.412109375 disk.dm-1.out‚ 234.83203125 disk.sda2.in‚0 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚131458.412109375 disk.sda3.in‚0 network.sent‚ 1.2370434e+07# disk.nvme0c0n1.in‚ 15721.8046875% disk.nvme0n1.out‚131458.412109375 disk.sda3.out‚ 234.83203125$ memory_percent‚96.34687900543213" proc.memory.rssMB‚ 3053.1328125* proc.memory.percent‚18.634843826293945 proc.cpu.threads‚17$ disk./.usageGB‚1177.788745880127 disk.dm-0.in‚ 15721.8046875 disk.dm-1.in‚0! disk.nvme0n1.in‚ 15721.8046875 disk.sda.in‚0 disk.sda.out‚ 234.83203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.9296875( proc.memory.availableMB‚ 598.52734375 cpu‚25.361010479369973>o¾“þ‚(:ˆ ìöëÔ¾÷õÏ gpu.0.gpu‚53.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸aMÔ{ÿ‚(:ð ìöëÔ¶ù¥à" proc.memory.rssMB‚ 3053.1328125 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚131597.349609375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.9296875 disk.sda.out‚ 235.171875* proc.memory.percent‚18.634843826293945 network.sent‚ 1.2374483e+07( disk./.usagePercent‚79.0431784457713 disk.dm-0.in‚15837.31640625 disk.sda2.out‚0% disk.nvme0n1.out‚131597.349609375 disk.sda3.in‚0( proc.memory.availableMB‚ 598.27734375 network.recv‚ 2.133211e+06$ disk./.usageGB‚1177.800479888916 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚131597.349609375" disk.nvme0n1.in‚15837.31640625$ memory_percent‚96.34840488433838 cpu‚25.372388716542474 proc.cpu.threads‚17 disk.dm-1.out‚ 235.171875 disk.sda2.in‚0$ disk.nvme0c0n1.in‚15837.31640625 disk.sda3.out‚ 235.171875 disk.sda.in‚0 û‘€‚(:† ûöëÔµÌ¼Ê gpu.0.gpu‚10.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Ü‘~‚(:ó ûöëÔä›çÎ disk.drbd1260.out‚ 0.93359375$ memory_percent‚96.34990692138672" proc.memory.rssMB‚ 3053.1328125% disk./.usageGB‚1177.6694450378418 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚15837.31640625% proc.memory.availableMB‚ 598.03125* proc.memory.percent‚18.634843826293945 cpu‚25.383919345731428 proc.cpu.threads‚17) disk./.usagePercent‚79.03438458697013% disk.nvme0n1.out‚131689.853515625 disk.sda.in‚0 disk.sda.out‚ 235.6328125 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚131689.853515625 network.sent‚ 1.2382635e+07 network.recv‚ 2.138127e+06" disk.nvme0n1.in‚15837.31640625 disk.sda3.in‚0" disk.dm-0.out‚131689.853515625 disk.dm-1.in‚0 disk.sda3.out‚ 235.6328125 disk.dm-0.in‚15837.31640625 disk.dm-1.out‚ 235.6328125 disk.sda2.in‚0 disk.sda2.out‚0"O$Õ“‚‚(:ˆ Š÷ëÔ”¾ŸÌ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚130.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šèÓ¹}ƒ‚(:ò Š÷ëÔøä¬Î disk.dm-1.out‚ 236.359375 disk.drbd1260.out‚ 0.93359375' disk.nvme0c0n1.out‚131834.435546875% disk.nvme0n1.out‚131834.435546875* proc.memory.percent‚18.634843826293945 cpu‚25.394852180402456% disk./.usageGB‚1177.6918449401855 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 236.359375$ memory_percent‚96.34687900543213( proc.memory.availableMB‚ 598.52734375" proc.memory.rssMB‚ 3053.1328125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚15837.31640625 disk.sda3.in‚0 disk.sda3.out‚ 236.359375 proc.cpu.threads‚17 network.sent‚ 1.2385731e+07( disk./.usagePercent‚79.0358878632112 disk.dm-0.in‚15837.31640625" disk.nvme0n1.in‚15837.31640625 disk.sda.in‚0 network.recv‚ 2.141646e+06" disk.dm-0.out‚131834.435546875ôwÜ2„- _wandbruntime‚6630  _runtime‚6630‚º[Ôa*…ÛT _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 122529080 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 223757793 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚60641209 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚407729452797952 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5792903752192 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚413272558575616 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚413272558575616 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 122549791 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5522365327590 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚29427503 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚399753734825894 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚394231369498304 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚407729452797952 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚90068932 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚449834400655360 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚394231352458944 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 101208000 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚407729470800896 % _timestamp‚1.7885418356080582e+09 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚218 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 trainer/global_step‚4835 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚407729470800896 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 101208002 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223757793 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 122549791 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚455627304407552 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚90068712 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚29427721 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚393 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚449834424595456 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚60641211 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚399753036719086Ÿ.è †àT E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 409073152 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚570 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 701042176 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5543087774720 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5793072156672 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚407729452797952 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚90068932 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚449834400655360 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚615 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3725 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚394231352458944 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 681067448 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚615 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1231 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 101208000 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚407729470800896 % _timestamp‚1.7885418356080582e+09 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 701042176 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1189768192 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚218 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 14135738368 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12553587712 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1185 trainer/global_step‚4835 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 13002869760 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1187728393 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚407729470800896 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 101208002 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 698106808 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13205766144 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223757793 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 122549791 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚455627304407552 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1185 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚90068712 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚29427721 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13205766144 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1195376640 _step‚393 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚449834424595456 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1185 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚570 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚60641211 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚399753036719086 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 122529080 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 223757793 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚60641209 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚407729452797952 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5792903752192 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 13002869760 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚413272558575616 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚413272558575616 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 122549791 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5522365327590 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 683039232 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚44 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚29427503 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚399753734825894 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚394231369498304 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14401142784 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 trainer/global_step‚4835 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223782083 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 _step‚394 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚413327309333504 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚407783713704960 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚399803951035154 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 101223478 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚455684254156800 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5793562007552 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5522172471026 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 101223476 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 122558605 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚413326608291328 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 192344576 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚449890692149248 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚413327309333504 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚407783713704960 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223761370 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 $ _timestamp‚1.788541836888309e+09 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5542894586368 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 122558605 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5793393603072 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1185 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚570 trainer/global_step‚4835 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 701042176 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚44 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1185 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3725 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223782083 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚570 _step‚394 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚413327309333504 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚407783713704960 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚399803951035154 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 13002869760 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20713 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12553587712 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 101223478 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚221 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 14135738368 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1268 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚455684254156800 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3725 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5793562007552 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 698106808 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 5522172471026 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 101223476 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 122558605 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚413326608291328 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 192344576 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚449890692149248 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1185 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚615 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚615 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚413327309333504 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚407783713704960 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223761370 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1195376640 $ _timestamp‚1.788541836888309e+09 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5542894586368 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 122558605 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1231 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5793393603072 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1185 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6396421790122986 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9734854698181152 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4232758581638336 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9775658845901489 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.021346915513277054 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9731569886207581 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9740623831748962 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9886198043823242 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2863648235797882 particle/accuracy_1_of_5‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.486287385225296 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.22721967101097107 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.364583283662796 ) Purity/*FRt*SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05818965658545494 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.061403512954711914 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3302468955516815 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5333333015441895 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6905932426452637 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.30575650930404663 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.35251009464263916 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3478808104991913 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 _runtime‚6650.706064579 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.23478488624095917 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3676127791404724 particle/accuracy_4_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8853427767753601 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.32510286569595337 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8611111044883728 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.219412699341774 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40864768624305725 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9596037864685059 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9820193648338318 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.979887843132019 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9829495549201965 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 *jet/accuracy_1_of_3‚0.8314962387084961 *jet/accuracy_1_of_5‚0.9263505339622498 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5015007853507996 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.009626216255128384 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2198924571275711 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9538689851760864 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24870464205741882 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9898791909217834 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028930816799402237 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.613095223903656 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +particle/specificity‚0.9500396251678467 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_7‚0 ) Purity/1FRt0SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03132345527410507 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03050595335662365 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333134651184 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3731409013271332 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.015105132944881916 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.544212818145752 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1591101884841919 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.450657457113266 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35944241285324097 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.22471341490745544 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5696645379066467 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3478808104991913 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.174502894282341 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 jet/accuracy_6_of_7‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.24557095766067505 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.19688110053539276 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2547297477722168 +jet/accuracy_3_of_4‚0.05843871086835861 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42959022521972656 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27657654881477356 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1688888967037201 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34500575065612793 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9479166865348816 particle/accuracy_3_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26518523693084717 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5289408564567566 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2953696548938751 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9934378862380981 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9919047951698303 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011231361888349056 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.38394469022750854 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.316091924905777 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_5_of_7‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25119051337242126 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1973787397146225 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5971872210502625 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2957058250904083 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_2_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 trainer/global_step‚4835 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.3087656497955322 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2220306098461151 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.21590416133403778 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004778760019689798 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9736394882202148 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333336114883423 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.383945494890213 *jet/accuracy_1_of_2‚0.7056237459182739 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.27656880021095276 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.25868523120880127 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.39188364148139954 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4463074803352356 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9846490621566772 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.25231480598449707 jet/accuracy_6_of_6‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.07051282376050949 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966135621070862 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2719925343990326 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016070907935500145 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2714994251728058 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35104164481163025 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2641414403915405 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976075887680054 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.031022584065794945 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8152785301208496 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.33183544874191284 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.22544090449810028 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.24696971476078033 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2321428507566452 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3243009150028229 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2132897824048996 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333134651184 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013359788805246353 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 particle/accuracy_6_of_7‚1 particle/accuracy_7_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006879736203700304 -jet/accuracy_4_of_4‚0.0011337868636474013 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2222222238779068 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3827519416809082 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5394604802131653 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1900263875722885 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3566901683807373 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3545386791229248 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8496610522270203 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1839657723903656 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9834277629852295 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.400086373090744 particle/accuracy_3_of_5‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09599298238754272 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5531550049781799 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01716378889977932 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07092198729515076 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3726631700992584 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0679059699177742 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2746647596359253 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3620690107345581 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5263677835464478 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.38562920689582825 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38377976417541504 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9827033281326294 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.079365074634552 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5972413420677185 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22821064293384552 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4014235734939575 ) Purity/*FRt0SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1395808309316635 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9839035272598267 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40940138697624207 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8819444179534912 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5418168306350708 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9605159163475037 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *jet/accuracy_1_of_4‚0.8728685975074768 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4333333671092987 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 _step‚395 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18003572523593903 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9747034311294556 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.013126210309565067 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2540138065814972 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.33181846141815186 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2796962559223175 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_5‚0.0853622630238533 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 )loss/val_total_loss‚45.75793189590357 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4150201976299286 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35728344321250916 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.8955027461051941 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4180986285209656 particle/accuracy_4_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4470957815647125 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4444444477558136 jet/accuracy_1_of_6‚1 ) Purity/1FRt*SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21861152350902557 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29200196266174316 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01331905834376812 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.33626091480255127 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48288407921791077 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_3_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9862555265426636 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22730696201324463 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966135621070862 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3535606861114502 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_7_of_7‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0013227513991296291 +jet/accuracy_2_of_5‚0.39746540784835815 particle/accuracy_1_of_2‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 jet/accuracy_2_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9848347306251526 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.20748135447502136 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 $ _timestamp‚1.788541855392554e+09 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9635828733444214 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.174502894282341 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09103564918041229 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.29942530393600464 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09724925458431244 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37478554248809814 *jet/accuracy_2_of_6‚0.3974359333515167 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8833333253860474 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31926432251930237 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851344108582 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175438523292542 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.40333378314971924 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21849842369556427 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3257310092449188 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5754258632659912 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454545319080353 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15922339260578156 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.567559540271759 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 * Purity/3FRt0SRqqt1FBt/FRt_purity‚0.375 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9133333563804626 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9605903625488281 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3480530381202698 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2679521143436432 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2162170112133026 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04179467260837555 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02935786545276642 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4309210479259491 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9989418387413025 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3944735527038574 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5941179990768433 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9889476299285889 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.33505693078041077 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28439879417419434 @&Pu‡Ü7nrity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9860103130340576 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6500457525253296 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.42445090413093567 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5324656367301941 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2668927311897278 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18419145047664642 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40611156821250916 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_4‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_7‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.34571489691734314”\ÍÈàwŽ¦ particle/accuracy_3_of_5‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09599298238754272 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5531550049781799 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01716378889977932 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.07092198729515076 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3726631700992584 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0679059699177742 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2746647596359253 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3620690107345581 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5263677835464478 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.38562920689582825 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38377976417541504 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9827033281326294 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.079365074634552 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5972413420677185 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22821064293384552 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4014235734939575 ) Purity/*FRt0SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1395808309316635 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9839035272598267 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.40940138697624207 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8819444179534912 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5418168306350708 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9605159163475037 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *jet/accuracy_1_of_4‚0.8728685975074768 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4333333671092987 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 _step‚395 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18003572523593903 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9747034311294556 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.013126210309565067 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2540138065814972 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.33181846141815186 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2796962559223175 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_5‚0.0853622630238533 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 )loss/val_total_loss‚45.75793189590357 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4150201976299286 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.35728344321250916 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.8955027461051941 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4180986285209656 particle/accuracy_4_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4470957815647125 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4444444477558136 jet/accuracy_1_of_6‚1 ) Purity/1FRt*SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21861152350902557 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29200196266174316 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01331905834376812 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.33626091480255127 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48288407921791077 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_3_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9862555265426636 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22730696201324463 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966135621070862 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3535606861114502 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_7_of_7‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0013227513991296291 +jet/accuracy_2_of_5‚0.39746540784835815 particle/accuracy_1_of_2‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 jet/accuracy_2_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9848347306251526 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.20748135447502136 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 $ _timestamp‚1.788541855392554e+09 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9635828733444214 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.174502894282341 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09103564918041229 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.29942530393600464 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09724925458431244 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.37478554248809814 *jet/accuracy_2_of_6‚0.3974359333515167 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8833333253860474 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.31926432251930237 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851344108582 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175438523292542 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.40333378314971924 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21849842369556427 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2708333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0052083334885537624 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.3257310092449188 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5754258632659912 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454545319080353 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15922339260578156 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.567559540271759 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 * Purity/3FRt0SRqqt1FBt/FRt_purity‚0.375 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9133333563804626 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9605903625488281 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3480530381202698 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2679521143436432 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2162170112133026 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04179467260837555 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02935786545276642 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4309210479259491 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9989418387413025 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3944735527038574 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5941179990768433 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9889476299285889 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.33505693078041077 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28439879417419434 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9860103130340576 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6500457525253296 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_2_of_5‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.42445090413093567 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5324656367301941 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2668927311897278 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18419145047664642 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40611156821250916 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_4_of_4‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_7‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.34571489691734314 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2563354969024658 particle/accuracy_6_of_6‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247461795807 jet/accuracy_5_of_6‚0 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9555555582046509 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3320852220058441 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9825581312179565 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9790069460868835 +particle/sensitivity‚0.3072880506515503 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2824074327945709 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2684723734855652 )jet/accuracy_2_of_4‚0.319441020488739 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3212643563747406 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9933122992515564 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9917412400245667 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.39994850754737854 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07902893424034119 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15427401661872864 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 'particle/f_score‚0.3894352912902832 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1477329134941101 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.01324140839278698 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.22274817526340485 epoch‚3 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2375660091638565 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.49431848526000977 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9923280477523804 +jet/accuracy_2_of_2‚0.10218563675880432 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6552082896232605 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3678258955478668 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_1_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6396421790122986 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9734854698181152 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4232758581638336 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9775658845901489 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.021346915513277054 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0729166716337204 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9731569886207581 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9740623831748962 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9886198043823242 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2863648235797882 particle/accuracy_1_of_5‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.486287385225296 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.22721967101097107 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.364583283662796 ) Purity/*FRt*SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05818965658545494 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.061403512954711914 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.3302468955516815 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5333333015441895 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6905932426452637 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.30575650930404663 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.35251009464263916 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3478808104991913 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 _runtime‚6650.706064579 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.23478488624095917 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_1‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3676127791404724 particle/accuracy_4_of_5‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3611111342906952 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8853427767753601 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.32510286569595337 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8611111044883728 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.219412699341774 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40864768624305725 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9596037864685059 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9820193648338318 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.979887843132019 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9829495549201965 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 *jet/accuracy_1_of_3‚0.8314962387084961 *jet/accuracy_1_of_5‚0.9263505339622498 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5015007853507996 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.009626216255128384 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2198924571275711 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9538689851760864 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24870464205741882 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9898791909217834 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028930816799402237 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.613095223903656 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +particle/specificity‚0.9500396251678467 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_7‚0 ) Purity/1FRt0SRqqt4FBt/FBt_purity‚0.75 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.03132345527410507 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03050595335662365 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3333333134651184 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3731409013271332 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.015105132944881916 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.544212818145752 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1591101884841919 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.450657457113266 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35944241285324097 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.22471341490745544 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5696645379066467 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3478808104991913 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.174502894282341 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 jet/accuracy_6_of_7‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.24557095766067505 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.19688110053539276 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2547297477722168 +jet/accuracy_3_of_4‚0.05843871086835861 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42959022521972656 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27657654881477356 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1688888967037201 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34500575065612793 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9479166865348816 particle/accuracy_3_of_6‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26518523693084717 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5289408564567566 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2953696548938751 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9934378862380981 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9919047951698303 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011231361888349056 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3611111342906952 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.38394469022750854 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.316091924905777 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 particle/accuracy_5_of_7‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25119051337242126 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1973787397146225 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5971872210502625 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2957058250904083 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_2_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 trainer/global_step‚4835 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.3087656497955322 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2220306098461151 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.21590416133403778 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004778760019689798 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 +Ш»ÚDjet/accuracy_4_of_6‚0.01923076994717121 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9736394882202148 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333336114883423 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.383945494890213 *jet/accuracy_1_of_2‚0.7056237459182739 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.27656880021095276 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735632538795471 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.25868523120880127 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.39188364148139954 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4463074803352356 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9846490621566772 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.25231480598449707 jet/accuracy_6_of_6‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.07051282376050949 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966135621070862 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2719925343990326 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.016070907935500145 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2714994251728058 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35104164481163025 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2641414403915405 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976075887680054 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.031022584065794945 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8152785301208496 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.33183544874191284 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.22544090449810028 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.24696971476078033 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2321428507566452 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3243009150028229 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2132897824048996 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3333333134651184 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013359788805246353 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 particle/accuracy_6_of_7‚1 particle/accuracy_7_of_7‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006879736203700304 -jet/accuracy_4_of_4‚0.0011337868636474013 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2222222238779068 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3827519416809082 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5394604802131653 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1900263875722885 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3566901683807373 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3545386791229248 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8496610522270203 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1839657723903656 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9834277629852295 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.400086373090744‹¿‚­“‚(:ˆ ¨÷ëÔ¿Ž³Ë gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.409744262695312* gpu.0.memoryAllocatedBytes‚ 14954856448 gpu.0.temp‚35.0 gpu.0.powerWatts‚167.517' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I‘Ч„‚(:ù ¨÷ëÔ…ÙŒÍ disk.sda3.out‚ 237.3046875 disk.sda.in‚0 disk.sda.out‚ 237.3046875# disk.dm-0.out‚133102.5849609375( disk./.usagePercent‚79.0637389228612 disk.sda2.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 237.3046875# proc.memory.rssMB‚ 3058.05859375* proc.memory.percent‚18.664908409118652 network.recv‚ 2.151613e+06$ disk./.usageGB‚1178.106845855713& disk.nvme0n1.out‚133102.5849609375 disk.sda1.in‚0$ memory_percent‚87.94548511505127 cpu‚25.407965843556674 network.sent‚ 1.2499048e+07" disk.nvme0n1.in‚15991.37109375 disk.dm-0.in‚15991.37109375 disk.dm-1.in‚0) proc.memory.availableMB‚ 1975.01171875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1260.out‚ 0.94140625$ disk.nvme0c0n1.in‚15991.37109375( disk.nvme0c0n1.out‚133102.5849609375zø|ÿ2‘- _wandbruntime‚6671  _runtime‚6671Ú[õÇ’Á % _timestamp‚1.7885418766330507e+09 _runtime‚6671.943995976 _step‚396 & lr-AdamW/pg1‚0.0012613771819898465 & lr-AdamW/pg2‚0.0012613771819898465 trainer/global_step‚4849*U¥™Ì“Æ _runtime‚6671.943995976 _step‚396 & lr-AdamW/pg1‚0.0012613771819898465 & lr-AdamW/pg2‚0.0012613771819898465 trainer/global_step‚4849 % _timestamp‚1.7885418766330507e+09Œæa× 2”- _wandbruntime‚6671  _runtime‚6671®ÃNù'•óO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415114705298432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5563623712256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395964546660592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14401142784 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101686746 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5815120131072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 925014364 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1187728393 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1185 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122973859 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415113777324032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 _runtime‚6671.945210505 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224660603 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123000203 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415113777324032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415114705298432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101686746 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409550171614720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60931924 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542824206550 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5564533683712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101686744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457610200697344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123000203 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451795104506368 $ _timestamp‚1.788541876634168e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401507370867142 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5563623712256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457610451322368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401508295881506 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90507596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5564533683712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451795080566272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60931922 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 trainer/global_step‚4849 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5815346816000 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101686744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224686949 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚44 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29576273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29575674 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543732181554 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 907975004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409550153611776 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚409550153611776 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415113777324032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3725 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250625024 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 353397248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13205766144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚615 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 927974400 _runtime‚6671.945210505 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 14135738368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1189768192 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1268 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚570 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚570 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224660603 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123000203 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1195376640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415113777324032 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚415114705298432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 14135738368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1185 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101686746 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409550171614720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 909971456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60931924 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542824206550 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1185 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚615 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1195376640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 13002869760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5564533683712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101686744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457610200697344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226684928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚615 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123000203 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451795104506368 $ _timestamp‚1.788541876634168e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401507370867142 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14401142784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5563623712256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12553587712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 184953856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14401142784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 909971456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457610451322368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1185 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401508295881506 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13684416521 _step‚397 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90507596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 927974400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5564533683712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1189768192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451795080566272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60931922 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3725 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344â î\2—- _wandbruntime‚6672  _runtime‚6672aÓ¸|8 ˜þM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415196517870080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚398 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚415197445857792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 927987712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 14135738368 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90525981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 184953856 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚615 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409631772697600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401588430749536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1195376640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409631772697600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1189768192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13205766144 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101706904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14401142784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409631790700544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254806016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5565655157248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1189768192 F