:W&BᾋܽM f87pgav1ª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0Odú‚" l5ak208cjthj f87pgav1ŠÖ f87pgav1juju11-uc-san-diegoSPAtop"™ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.2472920529188651 epochs‚10  hidden_dim‚256 ' learning_rate‚0.0006518520992777853  num_attention_heads‚4  num_branch_encoder_layers‚7  num_embedding_layers‚14  num_encoder_layers‚6Bjjtttt_blocks_f87pgav1bs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pŠ  ÄçÔ µ™‹š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64‚ šá7 f87pgav1&  _runtime‚0  _wandb.runtime‚0z«iÒã -Linux-6.8.0-138-generic-x86_64-with-glibc2.36CPython 3.12.10  ÄçÔà£ëD*-of*=/data/spatop/options_files/sweep_tttt_jj/config_f87pgav1.json*-l* /scratch/logs*-n*jjtttt_blocks_f87pgav12-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pr/opt/conda/bin/pythonº dpgn8w6v94cwjbe860ai8gw41lz7x4vz§“ÕZÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€À™ãü#ª€ äÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-8cb16037-0025-0eda-9425-82f6843a99c3Ê13.2º dpgn8w6v94cwjbe860ai8gw41lz7x4vzµª½Ç2  requirements.txtå0LuW f87pgav1ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64Å\øµY f87pgav1ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64ç¦5´*  f87pgav1b trainer/global_step2’Ñ8±\  f87pgav1ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64/ï;â/  f87pgav1b*"trainer/global_step2<¸ýŸK  f87pgav1j: ¡ÄçÔ觩Í*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ÕŸ«·’  f87pgav1j€  £ÄçÔØþ–Dî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 7.9 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 214 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ¶«”š[ f87pgav1jÉ £ÄçÔø¦ŽF·Trainable params: 226 M Non-trainable params: 86 Total params: 226 M Total estimated model params size (MB): 904.174 Modules in train mode: 5928 Modules in eval mode: 0 Total FLOPs: 0 ¸N±zp f87pgav1j_ £ÄçÔì›GPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ú6 Uz f87pgav1ji £ÄçÔ°¼ßGZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). /§[×x f87pgav1jg £ÄçÔ€‡öGXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). §×ò» f87pgav1jn £ÄçÔèÒ‰H_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). ¾]`Iy f87pgav1jh £ÄçÔøü•HYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). Óä s f87pgav1jb £ÄçÔ¸Ž¥HSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). ÄjVp f87pgav1j_ £ÄçÔ؆²HPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). n¦4Üs f87pgav1jb £ÄçÔðÎÁHSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). $e0'm f87pgav1j\ £ÄçÔ •ÏHMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ØkÓl f87pgav1j[ £ÄçÔèßØHLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). MO“"Ü f87pgav1*Ê  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"ö©ø8ê f87pgav1jØ £ÄçÔè­ý]È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. `^ ¼M f87pgav1j» £ÄçÔ ÒÌ^«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Çki(ë f87pgav1jÙ ©ÄçÔðö´·È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. â-zP f87pgav1j¾ ©ÄçÔè“È·­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. »½S¸‘‚(:‡ °ÄçÔø¤¤y gpu.0.gpu‚55.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.47273254394531* gpu.0.memoryAllocatedBytes‚ 33906819072 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EyÏAU‚(:Ë °ÄçÔŠÍ} disk.dm-1.in‚0) proc.memory.percent‚19.90485191345215) disk./.usagePercent‚77.25758170408773 disk.sda.out‚ 0.62109375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-1.out‚ 0.62109375 cpu‚12.827227727205536 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚33597 network.recv‚41498% disk./.usageGB‚1151.1937980651855 disk.drbd1260.out‚ 0.00390625 disk.sda1.in‚0 disk.nvme0c0n1.in‚38.78125% disk.nvme0c0n1.out‚385.1904296875 disk.dm-0.out‚385.1904296875$ memory_percent‚72.58450984954834) proc.memory.availableMB‚ 4491.75390625" proc.memory.rssMB‚ 3261.2109375 disk.sda.in‚0 disk.sda3.out‚ 0.62109375 disk.nvme0n1.in‚38.78125# disk.nvme0n1.out‚385.1904296875 disk.sda1.out‚0 disk.dm-0.in‚38.78125yb²  ‚(:† ¿ÄçÔöû¼x gpu.0.gpu‚41.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.49226379394531* gpu.0.memoryAllocatedBytes‚ 33923596288 gpu.0.temp‚38.0 gpu.0.powerWatts‚139.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.8812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÁ¯®Y!‚(:Ï ¿ÄçÔÆóùx disk.sda3.out‚ 1.1640625 disk.drbd1260.out‚ 0.00390625 disk.nvme0n1.in‚ 170.9296875 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 170.9296875( proc.memory.availableMB‚ 4489.8203125 network.recv‚46538 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 170.9296875 disk.dm-0.out‚731.3271484375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚43869) disk./.usagePercent‚77.26334571372115$ disk./.usageGB‚1151.279685974121# disk.nvme0n1.out‚731.3271484375% disk.nvme0c0n1.out‚731.3271484375 disk.sda.out‚ 1.1640625$ memory_percent‚72.59631156921387 disk.sda1.in‚0 disk.dm-1.out‚ 1.1640625" proc.memory.rssMB‚ 3261.9765625) proc.memory.percent‚19.90952491760254 cpu‚14.300331098095194 proc.cpu.threads‚17ƒÔâc"‚(:Ù ÎÄçÔ€…ªy disk.sda1.in‚0# proc.memory.rssMB‚ 3262.01171875 network.recv‚50173 disk.dm-0.in‚ 170.9296875" disk.dm-0.out‚1080.14404296875% disk.nvme0n1.out‚1080.14404296875 disk.sda2.in‚0$ memory_percent‚72.60012626647949 cpu‚15.462804752458437' disk.nvme0c0n1.out‚1080.14404296875 disk.dm-1.out‚ 1.80078125 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 network.sent‚49124) disk./.usagePercent‚77.27596509310646! disk.nvme0c0n1.in‚ 170.9296875 disk.sda.out‚ 1.80078125 disk.drbd1260.in‚0( proc.memory.availableMB‚ 4489.1953125) proc.memory.percent‚19.90973949432373 proc.cpu.threads‚17% disk./.usageGB‚1151.4677238464355 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 1.80078125 disk.drbd1260.out‚ 0.0078125 disk.nvme0n1.in‚ 170.9296875ÚÆ%8Ž#‚(:„ ÎÄçÔó´Žz gpu.0.gpu‚11.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.50447082519531* gpu.0.memoryAllocatedBytes‚ 33934082048 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.53' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Çü¿š$‚(: ÝÄçÔú²œy gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.50447082519531* gpu.0.memoryAllocatedBytes‚ 33934082048 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.711' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.942200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë1šnT%‚(:Ê ÝÄçÔÔîË{ disk.sda3.out‚2.3125 disk.nvme0n1.in‚ 170.9296875 disk.sda1.out‚0$ memory_percent‚72.61934280395508 proc.cpu.threads‚17' disk.nvme0c0n1.out‚1389.74951171875" disk.dm-0.out‚1389.74951171875 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 4486.046875 network.sent‚70607 network.recv‚80380) disk./.usagePercent‚77.30948595161729! disk.nvme0c0n1.in‚ 170.9296875 disk.dm-0.in‚ 170.9296875 disk.dm-1.out‚2.3125% disk.nvme0n1.out‚1389.74951171875( proc.memory.percent‚19.9099063873291% disk./.usageGB‚1151.9672088623047 disk.sda.out‚2.3125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.0078125 disk.sda1.in‚0" proc.memory.rssMB‚ 3262.0390625 cpu‚16.406204977666047Êkþ€š&‚(: ìÄçÔ¨¼x gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.50935363769531* gpu.0.memoryAllocatedBytes‚ 33938276352 gpu.0.temp‚38.0 gpu.0.powerWatts‚134.672' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.934399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿŸŒW'‚(:Í ìÄçÔÊ’Žz disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚72.61290550231934 cpu‚17.17758897250407 proc.cpu.threads‚17 network.sent‚74057 network.recv‚84137& disk.nvme0c0n1.out‚1748.5458984375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚2.671875 disk.sda3.out‚2.671875 disk.drbd1260.out‚ 0.01171875) disk./.usagePercent‚77.32626391738879$ disk./.usageGB‚1152.217212677002 disk.nvme0c0n1.in‚ 171.328125 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 171.328125$ disk.nvme0n1.out‚1748.5458984375 disk.sda1.in‚0( proc.memory.availableMB‚ 4487.1015625 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3262.05859375* proc.memory.percent‚19.910025596618652 disk.dm-0.in‚ 171.328125! disk.dm-0.out‚1748.5458984375 disk.dm-1.out‚2.671875ŒPåë_(‚(:Õ ûÄçÔ¼ÇÔy disk.sda1.in‚0 disk.drbd1260.out‚ 0.01171875$ disk.nvme0n1.out‚2167.8271484375 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚77.32468895903814 disk.dm-0.in‚ 205.5234375! disk.dm-0.out‚2167.8271484375 disk.dm-1.out‚ 3.1015625 disk.drbd1260.in‚0) proc.memory.availableMB‚ 4486.91015625 cpu‚17.82979377026102 network.recv‚90238 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 205.5234375& disk.nvme0c0n1.out‚2167.8271484375 disk.dm-1.in‚0 disk.sda3.out‚ 3.1015625 disk.nvme0n1.in‚ 205.5234375 disk.sda2.out‚0 network.sent‚81753% disk./.usageGB‚1152.1937446594238 disk.sda.in‚0 disk.sda.out‚ 3.1015625 disk.sda3.in‚0$ memory_percent‚72.61407375335693# proc.memory.rssMB‚ 3262.06640625* proc.memory.percent‚19.910073280334473l‰3´™)‚(: ûÄçԼƤz gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.50935363769531* gpu.0.memoryAllocatedBytes‚ 33938276352 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.255999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãìö3*b/ lr-AdamW/pg2"trainer/global_step2:JPhø[3+b/ lr-AdamW/pg1"trainer/global_step2:JPKZ-,) _wandbruntime‚95  _runtime‚95!_µ›À-» & lr-AdamW/pg2‚2.641915042565052e-05 trainer/global_step‚49 % _timestamp‚1.7884698891887238e+09 _runtime‚ 95.944825316 _step‚0 & lr-AdamW/pg1‚2.641915042565052e-05”ÀdÃÂ.½ & lr-AdamW/pg1‚2.641915042565052e-05 & lr-AdamW/pg2‚2.641915042565052e-05 trainer/global_step‚49 % _timestamp‚1.7884698891887238e+09 _runtime‚ 95.944825316 _step‚0ö¹«i/be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP]ùq0h0bd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPÄ[Þ i1be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP…0"àg2bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP”,'Ÿf3bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPëÔØØc4b_ l7bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPúVGjj8bf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP'!UÙf9bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP)_9ƒh:bd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPêÃ0fl;bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP{vÆnbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPôIÀi?be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPa²íšr@bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPâ“nAbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPÉ«ÞnBbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPÆwø¨jCbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP(PÀ#`Db\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP>)aEb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPÏÞòpFbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP—L{teGba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPD•ÚhHbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPðý¸ÔhIbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP=¤“{pJbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPÕK¼ŠqKbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPv«/_Lb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPå,òZkMbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPþ´¿u^NbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP6`µ?gObc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP¤›ePba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP„kx®oQbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPj“‹¼hRbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP»'”ÜnSbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP‹<ÄjTbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP/ù²fUbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPŽ , nVbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPŠ¼voWbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPÑÅùpXbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPrú‘RvYbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP‹|¾jZbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP˜fL¬s[bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPÒ •c\b_ ãsebo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÓÝïofbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPRÓ–$ggbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPÂßšDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP”Ý©Elnbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP³`Jiobe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPûVP`pb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP-† Qkqbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP·ná_rb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPÆ‹%^sbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPíþ-y]tbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP¤(Jùiube BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPßh4&svbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP¢ˆ½èfwbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPà "wmxbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP-i`)gybc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPЈV’nzbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP óŽk{bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPÜbfl|bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPOìÞUq}bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP• qi~be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPš9Ìrbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPãN‡xg€bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP›ïÀeb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPAдºc‚b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP¿]•tƒbo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP‰“B¶k„bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPR^5óy…bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP$¼ú‰`†b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPQ§—r‡bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPÆGvjˆbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP_†@b‰b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP¾íþ‘gŠbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JP›ç6Xh‹bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPTÎHmŒbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP2¸L«obj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP¡?’átŽbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP¬NÌhnbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPJo“hbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP=ÏÎÜp‘bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP™âŠ4q’bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPrU3¯f“ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPqCYl”bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPW»e•b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP°²Šq–bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP:pÜîn—bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP[¯\€g˜bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPôà Ht™bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP>I³ƒhšbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JP°|­bs›bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP'3ëhœbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JP§>V@ubp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPèËë"XžbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPtf¬}mŸbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP@tGQr bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP¡ˆý™j¡be BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP|¿o¢bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP>ñPo£bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP'#Áp¤bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPd]yîi¥bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPè–ju¦bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPJŠqf§ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPPˆl¨bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPYΗc©b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPqýÚ.ª) _wandbruntime‚95  _runtime‚95Üâ 5Ø«±O 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7687916179372 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30502 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759348 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 118118902428 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7854935474688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7013926682624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3714 trainer/global_step‚49 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1031895 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30502 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7736632630272 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3714 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1086644 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7854935474688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7684282096268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1246862 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1222020 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7736606238720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7566188621808 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2182 % _timestamp‚1.7884698891905012e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2253915 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2182 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 121939129856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2000 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1131 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509019136 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7858571760128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33384562688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2182 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1031895 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 169684991488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3714 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7858571760128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚444175 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 121727557564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 118329235968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33384562688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3714 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4575985664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7736632630272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1085939 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4575985664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 169616480768a&™ä»'¬µO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3714 trainer/global_step‚49 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1031895 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30502 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7736632630272 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3714 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1086644 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7854935474688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7684282096268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1246862 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1222020 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7736606238720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7566188621808 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2182 % _timestamp‚1.7884698891905012e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2253915 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2182 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 121939129856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2000 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1131 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509019136 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7858571760128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33384562688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2182 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1031895 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 169684991488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3714 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7858571760128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚444175 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 121727557564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 118329235968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33384562688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3714 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4575985664 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7736632630272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1085939 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4575985664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 169616480768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7687916179372 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30502 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759348 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 118118902428 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7854935474688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7013926682624n5h2f­ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP^R2m®bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP4W›¹m¯bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPú¬-Ôf°ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP‘^¶Èo±bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP‹²d²b_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPÊÔiRp½bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JPígéÝd¾b_ udÃb_ oÎbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPIW€rÏbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPõ;¤7jÐbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP³u9V`Ñb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPx’•†iÒbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP· ˜nÓbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP‰¹ƒkÔbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPLA¹shÕbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP-ç AjÖbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPqóòd×b_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPl÷(dÙb_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP¬{³ykçbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP¡©…zeèb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPœfÛféba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPpUõDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPmnÒr—bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÇ1 /g˜bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP?Ñri™bd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP¥`+pbšb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPSæÜBl›bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP¾ºÆ¡fœba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP¶¥sbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPf3Mmžbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPòMS{lŸbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP\m~k bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPÑCži¡bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPK/¹cj¢be BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPtõJö]£bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP÷ÓÚ}i¤bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPN}&g¥bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP%~g¦bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPDãÐMp§bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP¡é%.¨) _wandbruntime‚97  _runtime‚97V^`O©»M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1271455 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.curre·–=Ź#nt‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2323780 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7840651724002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2323780 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8014759814144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124349624832 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124349624832 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚654574 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7326191533568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68497408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7716492434976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3714 _step‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2182 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚654569 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33384562688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8014759814144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119700480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1052325 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2298906 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7890383797760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 124133861058 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3714 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2182 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 120739717632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1107683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 426574848 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7837017640690 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1052319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7153217071616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4575985664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1246587 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912žÑ÷½Å&ª¿M WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2323780 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7840651724002 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2323780 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8014759814144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124349624832 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124349624832 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚654574 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7326191533568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68497408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7716492434976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3714 _step‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2182 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚654569 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33384562688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8014759814144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119700480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1052325 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2298906 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7890383797760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 124133861058 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3714 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2182 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 120739717632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1107683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 426574848 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7837017640690 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1052319 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7153217071616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4575985664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1246587 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1271455 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613èêjA«b< loss/FRt3/assignment_loss"trainer/global_step2:JP_ÎêˆA¬b< loss/FBt1/assignment_loss"trainer/global_step2:JPiÖÌÔA­b< loss/FBt3/assignment_loss"trainer/global_step2:JPÂH3B®b= loss/SRqqt1/detection_loss"trainer/global_step2:JP¸ƒÑlB¯b= loss/SRqqt4/detection_loss"trainer/global_step2:JP-¼N-°b( epoch"trainer/global_step2:JPÃ]‰#@±b; loss/FBt1/detection_loss"trainer/global_step2:JPÀº@²b; loss/FBt3/detection_loss"trainer/global_step2:JPÂ|;ÒA³b< loss/FRt2/assignment_loss"trainer/global_step2:JPègA´b< loss/FRt4/assignment_loss"trainer/global_step2:JP ñÁ¯@µb; loss/FRt1/detection_loss"trainer/global_step2:JP-¾åÆB¶b= loss/SRqqt2/detection_loss"trainer/global_step2:JP îÿ7·b2 loss/total_loss"trainer/global_step2:JPœé²/C¸b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP‡ÕŸA¹b< loss/FBt2/assignment_loss"trainer/global_step2:JPðÉÍp@ºb; loss/FBt2/detection_loss"trainer/global_step2:JPÝ‚ŠôA»b< loss/FBt4/assignment_loss"trainer/global_step2:JP‚°ã]@¼b; loss/FRt2/detection_loss"trainer/global_step2:JPOï½C½b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP;YòîC¾b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPõ n#@¿b; loss/FRt3/detection_loss"trainer/global_step2:JPöü YCÀb> loss/SRqqt1/assignment_loss"trainer/global_step2:JP*Íï-AÁb< loss/FRt1/assignment_loss"trainer/global_step2:JPÕ@'(@Âb; loss/FRt4/detection_loss"trainer/global_step2:JPÔVBÃb= loss/SRqqt3/detection_loss"trainer/global_step2:JP§9 @Äb; loss/FBt4/detection_loss"trainer/global_step2:JPð5Ú.Å) _wandbruntime‚97  _runtime‚97ÿŸÜHÆ .loss/FBt2/detection_loss‚9.091126848757266 0loss/FBt4/assignment_loss‚2.6149276953477125 .loss/FRt2/detection_loss‚4.360112064370983 1loss/SRqqt3/assignment_loss‚5.494008612276903 1loss/SRqqt4/assignment_loss‚5.307705965198454 .loss/FRt3/detection_loss‚3.904216769066724 _runtime‚ 97.287338521 1loss/SRqqt1/assignment_loss‚5.476264430255425 % _timestamp‚1.7884698905314083e+09 /loss/FRt1/assignment_loss‚7.397751414775849 .loss/FRt4/detection_loss‚2.502565757417288 1loss/SRqqt3/detection_loss‚2.1648606974075535 .loss/FBt4/detection_loss‚1.964351098657429 /loss/FRt3/assignment_loss‚7.434198834679344 /loss/FBt1/assignment_loss‚2.662377192423894 0loss/FBt3/assignment_loss‚2.5935254096984863 0loss/SRqqt1/detection_loss‚4.273422070392749 0loss/SRqqt4/detection_loss‚2.510970205923573 epoch‚0 _step‚3 .loss/FBt1/detection_loss‚6.793586249534901 /loss/FBt3/detection_loss‚13.589209274603771 trainer/global_step‚49 /loss/FRt2/assignment_loss‚7.274538717771831 0loss/FRt4/assignment_loss‚7.1652329632493315 .loss/FRt1/detection_loss‚4.389821903407574 0loss/SRqqt2/detection_loss‚5.678973147273064 &loss/total_loss‚122.61169312867551 2loss/SRqqt2/assignment_loss‚5.4024908701578775 /loss/FBt2/assignment_loss‚2.565454936027527”0ÇLÇÆ .loss/FBt1/detection_loss‚6.793586249534901 /loss/FBt3/detection_loss‚13.589209274603771 trainer/global_step‚49 /loss/FRt2/assignment_loss‚7.274538717771831 0loss/FRt4/assignment_loss‚7.1652329632493315 .loss/FRt1/detection_loss‚4.389821903407574 0loss/SRqqt2/detection_loss‚5.678973147273064 &loss/total_loss‚122.61169312867551 2loss/SRqqt2/assignment_loss‚5.4024908701578775 /loss/FBt2/assignment_loss‚2.565454936027527 .loss/FBt2/detection_loss‚9.091126848757266 0loss/FBt4/assignment_loss‚2.6149276953477125 .loss/FRt2/detection_loss‚4.360112064370983 1loss/SRqqt3/assignment_loss‚5.494008612276903 1loss/SRqqt4/assignment_loss‚5.307705965198454 .loss/FRt3/detection_loss‚3.904216769066724 _runtime‚ 97.287338521 1loss/SRqqt1/assignment_loss‚5.476264430255425 % _timestamp‚1.7884698905314083e+09 /loss/FRt1/assignment_loss‚7.397751414775849 .loss/FRt4/detection_loss‚2.502565757417288 1loss/SRqqt3/detection_loss‚2.1648606974075535 .loss/FBt4/detection_loss‚1.964351098657429 /loss/FRt3/assignment_loss‚7.434198834679344 /loss/FBt1/assignment_loss‚2.662377192423894 0loss/FBt3/assignment_loss‚2.5935254096984863 0loss/SRqqt1/detection_loss‚4.273422070392749 0loss/SRqqt4/detection_loss‚2.510970205923573 epoch‚0 _step‚3l¢øü\È‚(:Ñ ŠÅçÔÒöøx disk.sda3.out‚ 3.43359375 cpu‚18.378522576983585 proc.cpu.threads‚17% disk./.usageGB‚1152.1938018798828 disk.nvme0n1.in‚ 287.640625 disk.sda2.out‚0 disk.sda.out‚ 3.43359375 disk.dm-0.in‚ 287.640625! disk.dm-0.out‚2549.5400390625 network.recv‚96331 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 287.640625 disk.dm-1.out‚ 3.43359375 disk.drbd1260.out‚0.015625& proc.memory.availableMB‚ 4485.90625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0$ memory_percent‚72.62020111083984# proc.memory.rssMB‚ 3262.09765625* proc.memory.percent‚19.910264015197754 network.sent‚93465) disk./.usagePercent‚77.32469279915115$ disk.nvme0n1.out‚2549.5400390625 disk.sda2.in‚0& disk.nvme0c0n1.out‚2549.5400390625³·Þ=É‚(:„ ŠÅçÔªÿ•y gpu.0.gpu‚35.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.51179504394531* gpu.0.memoryAllocatedBytes‚ 33940373504 gpu.0.temp‚39.0 gpu.0.powerWatts‚166.27' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ&)Ž’Ê‚(:‡ ™ÅçÔ²õ£x gpu.0.gpu‚71.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51179504394531* gpu.0.memoryAllocatedBytes‚ 33940373504 gpu.0.temp‚40.0 gpu.0.powerWatts‚127.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©õôé^Ë‚(:Ó ™ÅçÔùÏx disk.drbd1260.in‚0$ memory_percent‚72.62246608734131) proc.memory.availableMB‚ 4485.53515625" proc.memory.rssMB‚ 3262.1015625 cpu‚18.85330932254477) disk./.usagePercent‚77.35982317702974 disk.sda.in‚0 disk.drbd1260.out‚0.015625* proc.memory.percent‚19.910287857055664 proc.cpu.threads‚17$ disk./.usageGB‚1152.717269897461 disk.nvme0c0n1.in‚ 287.640625" disk.dm-0.out‚2993.02197265625 disk.dm-1.in‚0 disk.dm-1.out‚ 3.9765625 disk.sda3.in‚0 disk.nvme0n1.in‚ 287.640625% disk.nvme0n1.out‚2993.02197265625 network.sent‚143725' disk.nvme0c0n1.out‚2993.02197265625 disk.dm-0.in‚ 287.640625 disk.sda3.out‚ 3.9765625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚102599 disk.sda.out‚ 3.97656253ZŠ^\Ì‚(:Ñ ¨ÅçÔ‹åÐx disk.sda2.out‚0 disk.sda1.out‚0* proc.memory.percent‚19.910287857055664 network.sent‚149055% disk./.usageGB‚1152.7204055786133 disk.nvme0c0n1.in‚ 287.640625' disk.nvme0c0n1.out‚3151.41455078125" disk.dm-0.out‚3151.41455078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.01953125) proc.memory.availableMB‚ 4485.95703125 network.recv‚106270) disk./.usagePercent‚77.36003361522289 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚4.21875 disk.sda3.out‚4.21875% disk.nvme0n1.out‚3151.41455078125" proc.memory.rssMB‚ 3262.1015625 disk.dm-0.in‚ 287.640625 disk.sda1.in‚0$ memory_percent‚72.61989116668701 cpu‚19.269648544481253 proc.cpu.threads‚17 disk.dm-1.out‚4.21875 disk.sda3.in‚0 disk.nvme0n1.in‚ 287.640625 disk.sda2.in‚0ùoÑBÍ‚(:… ¨ÅçÔȬ‡y gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.51179504394531* gpu.0.memoryAllocatedBytes‚ 33940373504 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.3462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¨¡ _΂(:Ô ·ÅçÔŠŠz disk.sda1.out‚0 disk.nvme0c0n1.in‚ 287.640625" disk.dm-0.out‚3629.36376953125 proc.cpu.threads‚17% disk./.usageGB‚1152.6969184875488 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 3262.10546875* proc.memory.percent‚19.910311698913574 cpu‚19.63397715539219 network.sent‚156007 disk.sda.out‚4.671875 disk.sda3.in‚0 disk.sda3.out‚4.671875 disk.drbd1260.out‚ 0.01953125 disk.nvme0n1.in‚ 287.640625% disk.nvme0n1.out‚3629.36376953125) proc.memory.availableMB‚ 4485.90234375 network.recv‚112005) disk./.usagePercent‚77.35845737683456 disk.sda2.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚3629.36376953125 disk.dm-0.in‚ 287.640625 disk.dm-1.out‚4.671875$ memory_percent‚72.62022495269775èu*ˆ‘Ï‚(:† ·ÅçÔæÒ«z gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.51179504394531* gpu.0.memoryAllocatedBytes‚ 33940373504 gpu.0.temp‚39.0 gpu.0.powerWatts‚163.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áhP‘Ђ(:† ÆÅçÔúöÍy gpu.0.gpu‚59.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.51179504394531* gpu.0.memoryAllocatedBytes‚ 33940373504 gpu.0.temp‚39.0 gpu.0.powerWatts‚130.836' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã÷ŒçhÑ‚(:Ý ÆÅçԕ݆{ disk.sda2.out‚0$ memory_percent‚72.61841297149658) proc.memory.availableMB‚ 4486.19921875# proc.memory.rssMB‚ 3262.10546875 disk.dm-1.out‚ 5.23046875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 437.23828125 cpu‚19.954575754618705) disk./.usagePercent‚77.36003387123043 disk.sda.out‚ 5.23046875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 437.23828125! disk.dm-0.out‚3965.9248046875 network.sent‚163214 network.recv‚115982 disk.dm-1.in‚0 disk.sda3.out‚ 5.23046875 disk.sda1.out‚0& disk.nvme0c0n1.out‚3965.9248046875* proc.memory.percent‚19.910311698913574 proc.cpu.threads‚17% disk./.usageGB‚1152.7204093933105 disk.drbd1260.out‚ 0.0234375 disk.nvme0n1.in‚ 437.23828125$ disk.nvme0n1.out‚3965.9248046875;\ÇùÒ‚(:’ ÕÅçÔè·y gpu.0.gpu‚52.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51179504394531* gpu.0.memoryAllocatedBytes‚ 33940373504 gpu.0.temp‚40.0 gpu.0.powerWatts‚211.855' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.370999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b{DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2453722 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15425484190720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121810944 trainer/global_step‚99 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15664351923712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1282014 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832496640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70607872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 _step‚5 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4531665 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2017 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13978446132224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3715 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2453722 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚729 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3715 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4506817 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33386659840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15082625250640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33386659840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4578082816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15667988209152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2053101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2172668 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1282019 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14315169939456 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 238447657936 _runtime‚ 183.288215474 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4578082816 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 242477626880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2183 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 242477626880 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14315291750400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 336794415104 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 336723807232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2053101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33386659840 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2173397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816ÓÎè›Ï'ÙÉO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15425484190720 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2053095 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2453722 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15425484190720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121810944 trainer/global_step‚99 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15664351923712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1282014 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3832496640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70607872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 _step‚5 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4531665 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2017 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13978446132224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3715 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2453722 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚729 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3715 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4506817 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33386659840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15082625250640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33386659840 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4578082816 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15667988209152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2053101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2172668 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1282019 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14315169939456 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 238447657936 _runtime‚ 183.288215474 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4578082816 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 242477626880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2183 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 242477626880 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14315291750400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 336794415104 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 336723807232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2053101 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33386659840 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2173397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 238867732992 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2478564 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 238867732992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚891378 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15321072908576 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3715 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15667988209152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15425510582272 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2478564 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4578082816 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2183 % _timestamp‚1.7884699765306878e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13978497335296 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15082650678608 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚21833öå0Ú+ _wandbruntime‚184  _runtime‚184®þSÎÕ&ÛÏM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4576682 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 340077400064 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15473808452998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3715 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2183 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2073525 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832496640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15232954491776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2183 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4576682 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2194434 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14117736521216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1294824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15824176263168 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2478289 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2073525 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 244888121856 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4551808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15824176263168 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2478289 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 244888121856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4578082816 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2183 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4551808 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14457935718912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 184.563485345 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15579288141312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33386659840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4578082816 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 trainer/global_step‚99 % _timestamp‚1.7884699778068233e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15579288141312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚899615 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2017 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3401479946247÷1” ÜÓM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4578082816 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2183 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4551808 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14457935718912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 184.563485345 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15579288141312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33386659840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4578082816 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 trainer/global_step‚99 % _timestamp‚1.7884699778068233e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15579288141312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚899615 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2017 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 340147994624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4576682 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 340077400064 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15473808452998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3715 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2183 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2073525 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3832496640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15232954491776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2183 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4576682 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2194434 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14117736521216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1294824 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15824176263168 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2478289 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2073525 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 244888121856 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4551808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15824176263168 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2478289 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 244888121856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312IÀûH0Ý+ _wandbruntime‚184  _runtime‚184AV”¾JÞÄ _step‚7 0loss/FBt4/assignment_loss‚0.6459852935125431 0loss/FBt1/assignment_loss‚0.6633543566521257 _runtime‚ 184.566201556 .loss/FRt3/detection_loss‚3.417760072350502 .loss/FRt2/detection_loss‚4.315106483905212 1loss/SRqqt2/assignment_loss‚4.767032742500305 1loss/SRqqt1/detection_loss‚2.6941642357457067 0loss/SRqqt3/detection_loss‚4.248415393101306 .loss/FBt3/detection_loss‚4.581331886138234 /loss/FBt4/detection_loss‚3.7333551214084544 epoch‚0 trainer/global_step‚99 1loss/SRqqt1/assignment_loss‚4.790980515941497 1loss/SRqqt4/assignment_loss‚4.957365362267745 1loss/FBt2/assignment_loss‚0.35081813777963233 1loss/FBt3/assignment_loss‚0.12031141579417245 .loss/FBt1/detection_loss‚6.913752639293671 .loss/FBt2/detection_loss‚6.066969016300781 /loss/FRt1/detection_loss‚3.1387227204712955 /loss/FRt4/detection_loss‚2.3865153663791716 0loss/SRqqt2/detection_loss‚3.809199174696749 $loss/total_loss‚98.4409092803116 % _timestamp‚1.7884699778101616e+09 /loss/FRt2/assignment_loss‚7.269205798273501 0loss/FRt3/assignment_loss‚7.4426493740081785 .loss/FRt1/assignment_loss‚7.33929308977994 /loss/FRt4/assignment_loss‚7.203120991587639 0loss/SRqqt3/assignment_loss‚5.41987421201623 0loss/SRqqt4/detection_loss‚2.165625880407006n»¹FNßÈ 0loss/FBt4/assignment_loss‚0.6459852935125431 0loss/FBt1/assignment_loss‚0.6633543566521257 _runtime‚ 184.566201556 .loss/FRt3/detection_loss‚3.417760072350502 .loss/FRt2/detection_loss‚4.315106483905212 1loss/SRqqt2/assignment_loss‚4.767032742500305 1loss/SRqqt1/detection_loss‚2.6941642357457067 0loss/SRqqt3/detection_loss‚4.248415393101306 .loss/FBt3/detection_loss‚4.581331886138234 /loss/FBt4/detection_loss‚3.7333551214084544 epoch‚0 trainer/global_step‚99 1loss/SRqqt1/assignment_loss‚4.790980515941497 1loss/SRqqt4/assignment_loss‚4.957365362267745 1loss/FBt2/assignment_loss‚0.35081813777963233 1loss/FBt3/assignment_loss‚0.12031141579417245 .loss/FBt1/detection_loss‚6.913752639293671 .loss/FBt2/detection_loss‚6.066969016300781 /loss/FRt1/detection_loss‚3.1387227204712955 /loss/FRt4/detection_loss‚2.3865153663791716 0loss/SRqqt2/detection_loss‚3.809199174696749 $loss/total_loss‚98.4409092803116 % _timestamp‚1.7884699778101616e+09 /loss/FRt2/assignment_loss‚7.269205798273501 0loss/FRt3/assignment_loss‚7.4426493740081785 .loss/FRt1/assignment_loss‚7.33929308977994 /loss/FRt4/assignment_loss‚7.203120991587639 0loss/SRqqt3/assignment_loss‚5.41987421201623 0loss/SRqqt4/detection_loss‚2.165625880407006 _step‚7°Faà‚(:… äÅçÔ¥üÀy gpu.0.gpu‚16.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.51179504394531* gpu.0.memoryAllocatedBytes‚ 33940373504 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.315' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.463 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çìT-aá‚(:Ö äÅçÔ彩% disk.nvme0c0n1.out‚4658.212890625 disk.dm-0.out‚4658.212890625' proc.memory.availableMB‚ 4487.046875 network.sent‚173448 disk.sda3.in‚0 disk.drbd1260.in‚0! proc.memory.rssMB‚ 3262.109375* proc.memory.percent‚19.910335540771484 cpu‚20.498889312384737 proc.cpu.threads‚17 network.recv‚125093# disk.nvme0n1.out‚4658.212890625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 6.08984375$ memory_percent‚72.61323928833008 disk.sda3.out‚ 6.08984375 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 437.23828125 disk.dm-0.in‚ 437.23828125 disk.dm-1.out‚ 6.08984375 disk.sda.in‚0) disk./.usagePercent‚77.36161292570162$ disk./.usageGB‚1152.743938446045 disk.drbd1260.out‚ 0.02734375 disk.nvme0n1.in‚ 437.23828125 disk.sda1.out‚0ù+–’Vâ‚(:Ë óÅçÔ߆­x* proc.memory.percent‚19.910335540771484 proc.cpu.threads‚17) disk./.usagePercent‚77.36161164566394 disk.dm-0.in‚ 437.23828125 disk.dm-0.out‚4956.318359375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 cpu‚20.731892870308506 network.recv‚131966 disk.dm-1.out‚6.5625 disk.sda3.out‚6.5625 disk.nvme0n1.in‚ 437.23828125 disk.sda1.out‚0% disk.nvme0c0n1.out‚4956.318359375' proc.memory.availableMB‚ 4487.015625 network.sent‚223960% disk./.usageGB‚1152.7439193725586 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.02734375# disk.nvme0n1.out‚4956.318359375 disk.sda2.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 437.23828125 disk.sda.out‚6.5625 disk.sda2.out‚0$ memory_percent‚72.61343002319336! proc.memory.rssMB‚ 3262.109375”<¯‘ã‚(:† óÅçÔÏŸÈy gpu.0.gpu‚48.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚38.0 gpu.0.powerWatts‚131.362' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.2724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡gÆ®`ä‚(:Õ ‚ÆçÔéοx* proc.memory.percent‚19.910335540771484$ disk./.usageGB‚1152.743938446045 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 6.9453125 disk.drbd1260.out‚0.03125 disk.nvme0n1.in‚ 552.41015625 disk.sda2.in‚0 disk.sda1.in‚0 network.sent‚229230 disk.sda1.out‚0& disk.nvme0c0n1.out‚5303.8857421875 disk.dm-0.in‚ 552.41015625! disk.dm-0.out‚5303.8857421875$ memory_percent‚72.61340618133545) proc.memory.availableMB‚ 4487.01953125! proc.memory.rssMB‚ 3262.109375) disk./.usagePercent‚77.36161292570162 disk.dm-1.out‚ 6.9453125 disk.sda.out‚ 6.9453125 disk.sda3.in‚0$ disk.nvme0n1.out‚5303.8857421875" disk.nvme0c0n1.in‚ 552.41015625 cpu‚20.943735474326402 proc.cpu.threads‚17 network.recv‚135559 disk.drbd1260.in‚0 disk.sda2.out‚0C*ªh’å‚(:‡ ‚ÆçÔÇŒ¨y gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚44.0 gpu.0.powerWatts‚239.923' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.9846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Ø8dæ‚(:Ù ‘ÆçÔ‡Åùx disk.drbd1260.in‚0$ memory_percent‚72.61202335357666! proc.memory.rssMB‚ 3262.109375 cpu‚21.137373190465855 proc.cpu.threads‚17 network.recv‚140848) disk./.usagePercent‚77.36003847936604% disk./.usageGB‚1152.7204780578613 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 552.41015625& disk.nvme0c0n1.out‚5554.9560546875! disk.dm-0.out‚5554.9560546875 disk.dm-1.in‚0 network.sent‚235816 disk.nvme0n1.in‚ 552.41015625 disk.dm-1.out‚ 7.42578125 disk.sda.out‚ 7.42578125 disk.sda3.out‚ 7.42578125 disk.drbd1260.out‚0.03125* proc.memory.percent‚19.910335540771484$ disk.nvme0n1.out‚5554.9560546875 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 552.41015625 disk.sda.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 4487.24609375 disk.sda1.in‚0)&ø“‘ç‚(:† ‘ÆçÔí’ëy gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÙœè‚(:‘  ÆçÔÓ§¡x gpu.0.gpu‚35.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚44.0 gpu.0.powerWatts‚281.998' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.39959999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23473768372736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21446380072448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21446256164352 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22599087307440 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3710266 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23114388534272 _step‚9 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚731 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3074301 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 362385068580 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4580179968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3258979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23473768372736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3074295 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3259710 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3716 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 503290582528 $ _timestamp‚1.788470057446542e+09 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3074295 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3716 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 363016123904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1922269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 359406230016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4580179968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 484820992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23477404658176 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 359406230016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4580179968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3258979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚23473768372736 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3074295 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3259710 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3716 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 503290582528 $ _timestamp‚1.788470057446542e+09 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3074295 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3716 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 363016123904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1922269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 359406230016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4580179968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 484820992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚23477404658176 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 359406230016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚23473768372736 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21446380072448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21446256164352 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22599087307440 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3710266 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23114388534272 _step‚9 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚731 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3074301 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 362385068580 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23268166093312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1935074 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6829584 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4580179968 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21589007388672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23114233348618 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884700585466614e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 365426618880 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22749441976544 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6804710 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6829584 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6804710 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 trainer/global_step‚149 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3280727 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3094725 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23110599265306 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22749416548576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23268139701760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 484820992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3094719 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 364791372074 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4580179968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚714 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3849273856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3716 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23268139701760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3716 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3734859 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3734859 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21082307173888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚23633592712192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3094719 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33388756992 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3094725 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚23629956413440 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 365426618880 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 361816711680 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3709991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3716 _step‚10 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23268166093312 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1935074 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6829584 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4580179968 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21589007388672 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23114233348618 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884700585466614e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 365426618880 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22749441976544 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6804710 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6829584 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6804710 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 trainer/global_step‚149 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3280727 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3094725 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23110599265306 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22749416548576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23268139701760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 484820992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3094719 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 364791372074 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4580179968 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚714 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3849273856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3716 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23268139701760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3716 K;0ó+ _wandbruntime‚265  _runtime‚265™…TôÎ 1loss/SRqqt2/detection_loss‚13.813918247818947 1loss/SRqqt3/detection_loss‚6.1602467075725995 1loss/SRqqt4/detection_loss‚1.6409048621471112 .loss/FBt2/detection_loss‚8.088050280298505 .loss/FBt4/detection_loss‚4.244974539304782 .loss/FRt4/assignment_loss‚7.21967022297746 _step‚11 /loss/FRt3/assignment_loss‚7.294981549425823 1loss/SRqqt2/assignment_loss‚5.370407581329346 0loss/FBt1/assignment_loss‚0.6184426463959325 % _timestamp‚1.7884700585493758e+09 _runtime‚ 265.305287179 &loss/total_loss‚121.17628633376782 epoch‚0 2loss/SRqqt1/assignment_loss‚3.6515828098569596 2loss/SRqqt3/assignment_loss‚5.4570662677288055 /loss/FRt1/detection_loss‚3.5717053340405833 /loss/FRt4/detection_loss‚2.7387106015773144 -loss/FBt3/detection_loss‚2.27022289658158 trainer/global_step‚149 0loss/FRt2/assignment_loss‚7.0307002348058365 /loss/FRt2/detection_loss‚3.1552339652005363 /loss/FBt1/detection_loss‚19.594646553198498 0loss/FRt1/assignment_loss‚7.3945503332176985 2loss/SRqqt4/assignment_loss‚3.4007780447111027 1loss/FBt3/assignment_loss‚0.30681779191694786 0loss/SRqqt1/detection_loss‚2.353571540755885 0loss/FBt2/assignment_loss‚0.4419190408634244 /loss/FBt4/assignment_loss‚1.159454916914304 /loss/FRt3/detection_loss‚4.1977293651278424£¦PXõÒ trainer/global_step‚149 0loss/FRt2/assignment_loss‚7.0307002348058365 /loss/FRt2/detection_loss‚3.1552339652005363 /loss/FBt1/detection_loss‚19.594646553198498 0loss/FRt1/assignment_loss‚7.3945503332176985 2loss/SRqqt4/assignment_loss‚3.4007780447111027 1loss/FBt3/assignment_loss‚0.30681779191694786 0loss/SRqqt1/detection_loss‚2.353571540755885 0loss/FBt2/assignment_loss‚0.4419190408634244 /loss/FBt4/assignment_loss‚1.159454916914304 /loss/FRt3/detection_loss‚4.1977293651278424 1loss/SRqqt2/detection_loss‚13.813918247818947 1loss/SRqqt3/detection_loss‚6.1602467075725995 1loss/SRqqt4/detection_loss‚1.6409048621471112 .loss/FBt2/detection_loss‚8.088050280298505 .loss/FBt4/detection_loss‚4.244974539304782 .loss/FRt4/assignment_loss‚7.21967022297746 _step‚11 /loss/FRt3/assignment_loss‚7.294981549425823 1loss/SRqqt2/assignment_loss‚5.370407581329346 0loss/FBt1/assignment_loss‚0.6184426463959325 % _timestamp‚1.7884700585493758e+09 _runtime‚ 265.305287179 &loss/total_loss‚121.17628633376782 epoch‚0 2loss/SRqqt1/assignment_loss‚3.6515828098569596 2loss/SRqqt3/assignment_loss‚5.4570662677288055 /loss/FRt1/detection_loss‚3.5717053340405833 /loss/FRt4/detection_loss‚2.7387106015773144 -loss/FBt3/detection_loss‚2.27022289658158 W”Žâö‚(:’ ¯ÆçÔª²x gpu.0.gpu‚97.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚43.0 gpu.0.powerWatts‚120.874' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.174799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÏŒ¡Z÷‚(:Ï ¯ÆçÔØ¥¼y! disk.dm-0.out‚6262.5419921875 disk.sda.in‚0 disk.sda.out‚8.40625$ disk.nvme0n1.out‚6262.5419921875 network.sent‚247917 network.recv‚149952% disk./.usageGB‚1151.3678588867188 disk.dm-1.out‚8.40625 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 3262.109375 disk.dm-1.in‚0 disk.sda3.out‚8.40625 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.03515625 disk.nvme0n1.in‚ 552.41015625 disk.sda2.in‚0$ memory_percent‚72.61419296264648' proc.memory.availableMB‚ 4486.890625* proc.memory.percent‚19.910335540771484 cpu‚21.4709350920629 proc.cpu.threads‚17) disk./.usagePercent‚77.26926307186758" disk.nvme0c0n1.in‚ 552.41015625& disk.nvme0c0n1.out‚6262.5419921875 disk.dm-0.in‚ 552.41015625¹º¤‚’ø‚(:‡ ¾ÆçÔ’þÂw gpu.0.gpu‚41.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚48.0 gpu.0.powerWatts‚274.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D‚Ø’gù‚(:Ü ¾ÆçÔÑßµx) disk./.usagePercent‚77.21748554806749 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 737.02734375 disk.dm-0.in‚ 737.02734375" disk.dm-0.out‚6518.01806640625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.0390625 network.sent‚297074% disk.nvme0n1.out‚6518.01806640625 disk.sda2.in‚0 disk.dm-1.out‚ 10.1328125$ memory_percent‚72.61435985565186* proc.memory.percent‚19.910335540771484 disk.nvme0n1.in‚ 737.02734375 disk.dm-1.in‚0 disk.sda3.out‚ 10.1328125) proc.memory.availableMB‚ 4486.86328125 cpu‚21.61128727803604 proc.cpu.threads‚17$ disk./.usageGB‚1150.596336364746 disk.sda2.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚6518.01806640625 disk.sda.in‚0 disk.sda.out‚ 10.1328125 disk.drbd1260.in‚0! proc.memory.rssMB‚ 3262.109375 network.recv‚154981Ýøgú‚(:Ü ÍÆçÔÑÙõy) proc.memory.availableMB‚ 4486.35546875 network.recv‚159784 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 737.02734375 disk.sda.in‚0 disk.sda.out‚ 10.6640625 disk.drbd1260.in‚0! proc.memory.rssMB‚ 3262.109375) disk./.usagePercent‚77.21748452403735$ disk./.usageGB‚1150.596321105957" disk.nvme0c0n1.in‚ 737.02734375" disk.dm-0.out‚6687.52978515625 disk.sda3.in‚0* proc.memory.percent‚19.910335540771484 cpu‚21.74387965905276 proc.cpu.threads‚17 disk.nvme0n1.in‚ 737.02734375' disk.nvme0c0n1.out‚6687.52978515625 disk.dm-1.out‚ 10.6640625 disk.sda3.out‚ 10.6640625$ memory_percent‚72.61745929718018 network.sent‚303544% disk.nvme0n1.out‚6687.52978515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.0390625oö—û‚(:’ ÍÆçÔÓÑßz gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚47.0 gpu.0.powerWatts‚284.214' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.842800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈJÀ‘ü‚(:† ÜÆçÔÃÙÕw gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚43.0 gpu.0.powerWatts‚127.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.519 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RJdý‚(:Ù ÜÆçÔÒÑèx) disk./.usagePercent‚77.21748093993187" disk.nvme0n1.out‚ 6953.67578125 disk.sda2.in‚0 disk.dm-0.in‚ 737.02734375 disk.sda.in‚0 disk.sda3.out‚ 11.27734375$ memory_percent‚72.61593341827393 disk.drbd1260.in‚0 disk.drbd1260.out‚0.046875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 737.02734375$ disk.nvme0c0n1.out‚ 6953.67578125 disk.dm-0.out‚ 6953.67578125) proc.memory.availableMB‚ 4486.60546875* proc.memory.percent‚19.910359382629395 disk.sda.out‚ 11.27734375 disk.sda3.in‚0 cpu‚21.865247231255413 proc.cpu.threads‚17 network.sent‚310659% disk./.usageGB‚1150.5962677001953 disk.nvme0n1.in‚ 737.02734375 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 11.27734375# proc.memory.rssMB‚ 3262.11328125 network.recv‚163300kû0þ+ _wandbruntime‚324  _runtime‚324ä¥ñYÆÿÀ _runtime‚ 324.597170079 _step‚12 ' lr-AdamW/pg1‚0.00010729410070825415 ' lr-AdamW/pg2‚0.00010729410070825415 trainer/global_step‚199 % _timestamp‚1.7884701178404224e+09f¯§eÊ€Ä % _timestamp‚1.7884701178404224e+09 _runtime‚ 324.597170079 _step‚12 ' lr-AdamW/pg1‚0.00010729410070825415 ' lr-AdamW/pg2‚0.00010729410070825415 trainer/global_step‚199 C·„[0+ _wandbruntime‚324  _runtime‚324Eˆé:ö‚ÉO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30803266486272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocak]ÎsÙ&tions.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30598288616296 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33388756992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28577491589632 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1781657 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 324.600526969 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30115574792208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4095495 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 479944727040 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9037469 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3716 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3716 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31286821107200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 486790144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33388756992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4580179968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2184 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9012621 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 479105168952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27907536234496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 483554620928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 479944727040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2562514 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 trainer/global_step‚199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 670006558208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 670075068928 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4580179968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4580179968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2184 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4917126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30803266486272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30803240094720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3716 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30803240094720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1782337 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3849273856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4095501 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 482713824088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28577611303424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4941968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31286821107200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4095501 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31283184821760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 483554620928 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4095495 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4941968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3716 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4344856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33388756992 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2562519 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31283184821760 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33388756992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4580179968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2184 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9012621 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 479105168952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27907536234496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 483554620928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 479944727040 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2562514 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 trainer/global_step‚199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 670006558208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 670075068928 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4580179968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4580179968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2184 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4917126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30803266486272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30803240094720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3716 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30803240094720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1782337 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3849273856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4095501 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 482713824088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28577611303424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4941968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31286821107200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4095501 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31283184821760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 483554620928 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4095495 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4941968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3716 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4344856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 .ðÒÖ0„+ _wandbruntime‚326  _runtime‚326c¼îÙ&…ÓM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 482355208704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68497408 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 673325029888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4366498 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 673393527296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119700480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30957044045312 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4966561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3849273856 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33388756992 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30747390077614 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4580179968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9057612 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28720220150784 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30957044045312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30265904033344 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30265878605376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33388756992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3716 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28720100450304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4941693 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1791174 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 485965115904 trainer/global_step‚199 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9057612 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 481511472238 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3716 _runtime‚ 326.360576719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4365838 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28046826623488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4115925 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 486790144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33388756992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2575319 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4580179968 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4115919 % _timestamp‚1.7884701196045535e+09 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4115925 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2575324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4580179968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2184 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31443009161216 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30751024160926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3716 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31439372862464 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9082486 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0L {¸( †×M R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31439372862464 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9082486 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 482355208704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68497408 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 673325029888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4366498 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 673393527296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119700480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30957044045312 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4966561 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3849273856 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33388756992 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30747390077614 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4580179968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9057612 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28720220150784 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30957044045312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30265904033344 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30265878605376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33388756992 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3716 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28720100450304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4941693 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1791174 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 485965115904 trainer/global_step‚199 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9057612 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 481511472238 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3716 _runtime‚ 326.360576719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4365838 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28046826623488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4115925 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 486790144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33388756992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2575319 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4580179968 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4115919 % _timestamp‚1.7884701196045535e+09 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4115925 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2575324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4580179968 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2184 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31443009161216 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30751024160926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3716íYm0‡+ _wandbruntime‚326  _runtime‚326èɘRˆÌ epoch‚0 0loss/FBt1/assignment_loss‚0.7043624492362142 0loss/FBt4/assignment_loss‚0.6658092141151428 trainer/global_step‚199 /loss/FBt2/detection_loss‚10.350524733774364 .loss/FBt4/detection_loss‚12.82701397324837 &loss/total_loss‚110.17922591520633 % _timestamp‚1.7884701196062791e+09 .loss/FRt1/detection_loss‚4.778147860947582 1loss/SRqqt1/detection_loss‚3.4339046542551004 /loss/FRt1/assignment_loss‚7.383061899079217 /loss/FRt3/assignment_loss‚7.427083073240338 2loss/SRqqt1/assignment_loss‚3.3103100084790995 1loss/SRqqt3/assignment_loss‚4.485594298158373 0loss/FBt2/assignment_loss‚0.6020204348969855 _runtime‚ 326.361642088 /loss/FRt2/assignment_loss‚7.370035343170166 -loss/FRt3/detection_loss‚5.01987352560867 .loss/FBt3/detection_loss‚1.862155230175103 1loss/FBt3/assignment_loss‚0.33245646434267156 .loss/FRt4/detection_loss‚2.285819314531426 1loss/SRqqt3/detection_loss‚3.5057046452906357 2loss/SRqqt2/assignment_loss‚3.4729328599087026 2loss/SRqqt4/assignment_loss‚3.4691359996795654 .loss/FRt2/detection_loss‚4.151980739831925 1loss/SRqqt2/detection_loss‚3.5863817001498024 1loss/SRqqt4/detection_loss‚1.5286359809641097 /loss/FBt1/detection_loss‚10.446061527356505 _step‚15 /loss/FRt4/assignment_loss‚7.180219984766262‚Ðb V‰Ð 2loss/SRqqt2/assignment_loss‚3.4729328599087026 2loss/SRqqt4/assignment_loss‚3.4691359996795654 .loss/FRt2/detection_loss‚4.151980739831925 1loss/SRqqt2/detection_loss‚3.5863817001498024 1loss/SRqqt4/detection_loss‚1.5286359809641097 /loss/FBt1/detection_loss‚10.446061527356505 _step‚15 /loss/FRt4/assignment_loss‚7.180219984766262 epoch‚0 0loss/FBt1/assignment_loss‚0.7043624492362142 0loss/FBt4/assignment_loss‚0.6658092141151428 trainer/global_step‚199 /loss/FBt2/detection_loss‚10.350524733774364 .loss/FBt4/detection_loss‚12.82701397324837 &loss/total_loss‚110.17922591520633 % _timestamp‚1.7884701196062791e+09 .loss/FRt1/detection_loss‚4.778147860947582 1loss/SRqqt1/detection_loss‚3.4339046542551004 /loss/FRt1/assignment_loss‚7.383061899079217 /loss/FRt3/assignment_loss‚7.427083073240338 2loss/SRqqt1/assignment_loss‚3.3103100084790995 1loss/SRqqt3/assignment_loss‚4.485594298158373 0loss/FBt2/assignment_loss‚0.6020204348969855 _runtime‚ 326.361642088 /loss/FRt2/assignment_loss‚7.370035343170166 -loss/FRt3/detection_loss‚5.01987352560867 .loss/FBt3/detection_loss‚1.862155230175103 1loss/FBt3/assignment_loss‚0.33245646434267156 .loss/FRt4/detection_loss‚2.285819314531426 1loss/SRqqt3/detection_loss‚3.5057046452906357‡c©…šŠ‚(: ëÆçÔð¡áx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.878' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.97559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-ùÚa‹‚(:Ö ëÆçÔŸöÉz disk.sda.out‚ 11.7890625 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 737.02734375 cpu‚21.984809659825903 disk.dm-1.out‚ 11.7890625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.046875 disk.sda2.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 737.02734375$ memory_percent‚72.61660099029541) proc.memory.availableMB‚ 4486.49609375 network.recv‚168695) disk./.usagePercent‚77.21069085210121 disk.sda.in‚0 disk.sda3.out‚ 11.7890625$ disk.nvme0c0n1.out‚ 7027.24609375 disk.dm-0.out‚ 7027.24609375* proc.memory.percent‚19.910359382629395 network.sent‚315575 disk.nvme0n1.in‚ 737.02734375" disk.nvme0n1.out‚ 7027.24609375 disk.sda1.in‚0# proc.memory.rssMB‚ 3262.11328125 proc.cpu.threads‚17% disk./.usageGB‚1150.4950904846191 disk.dm-1.in‚0)’’Œ‚(:‡ úÆçÔ‘²Çy gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚47.0 gpu.0.powerWatts‚275.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08° R`‚(:Õ úÆçÔ’íûy) disk./.usagePercent‚77.21614918873875 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 885.91015625 disk.dm-0.in‚ 885.91015625 disk.drbd1260.in‚0 disk.nvme0n1.out‚ 7377.328125 disk.sda2.out‚0" disk.nvme0c0n1.out‚ 7377.328125 disk.dm-0.out‚ 7377.328125 disk.sda.out‚ 12.17578125 disk.sda3.out‚ 12.17578125 disk.drbd1260.out‚ 0.05078125* proc.memory.percent‚19.910359382629395 network.recv‚172908 disk.nvme0n1.in‚ 885.91015625 disk.dm-1.in‚0 disk.dm-1.out‚ 12.17578125 disk.sda3.in‚0$ memory_percent‚72.61507511138916) proc.memory.availableMB‚ 4486.74609375 cpu‚22.087416043535743% disk./.usageGB‚1150.5764236450195 disk.sda.in‚0# proc.memory.rssMB‚ 3262.11328125 proc.cpu.threads‚17 network.sent‚363673 O‘Ž‚(:† ‰ÇçÔù›x gpu.0.gpu‚55.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚41.0 gpu.0.powerWatts‚189.396' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.8792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ^zuc‚(:Ø ‰ÇçÔ¸ã©y disk.nvme0n1.in‚ 885.91015625" disk.nvme0n1.out‚ 7557.94921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 885.91015625$ disk.nvme0c0n1.out‚ 7557.94921875 disk.dm-1.out‚ 12.62890625 disk.sda3.out‚ 12.62890625 disk.drbd1260.out‚ 0.05078125) proc.memory.availableMB‚ 4486.73046875 cpu‚22.18875575991267 network.sent‚370320 disk.sda2.in‚0 disk.dm-0.out‚ 7557.94921875 disk.dm-1.in‚0 disk.sda.out‚ 12.62890625# memory_percent‚72.6151704788208# proc.memory.rssMB‚ 3262.11328125* proc.memory.percent‚19.910359382629395 network.recv‚178010 disk.dm-0.in‚ 885.91015625 disk.drbd1260.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚77.22159984515025$ disk./.usageGB‚1150.657642364502•Š+òa‚(:Ö ˜ÇçÔš‰ªy network.recv‚181835% disk./.usageGB‚1150.6576118469238" disk.nvme0c0n1.in‚ 885.91015625 disk.sda.in‚0 disk.sda.out‚ 13.04296875 disk.sda3.in‚0 disk.sda3.out‚ 13.04296875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 885.91015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 7750.7890625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.0546875$ memory_percent‚72.61700630187988 network.sent‚375414) disk./.usagePercent‚77.22159779708998 disk.dm-1.out‚ 13.04296875( proc.memory.availableMB‚ 4486.4296875# proc.memory.rssMB‚ 3262.11328125* proc.memory.percent‚19.910359382629395 proc.cpu.threads‚17 disk.dm-0.out‚ 7750.7890625! disk.nvme0n1.out‚ 7750.7890625 disk.sda1.out‚0 disk.dm-0.in‚ 885.91015625 cpu‚22.285704966689252€g¢¦‘‘‚(:† ˜ÇçÔŸ¯ôy gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚41.0 gpu.0.powerWatts‚231.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.2698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á‹DV’’‚(:‡ §ÇçÔÅÚ¢w gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.738' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü|WNa“‚(:Ö §ÇçÔÚú¶x network.sent‚382727( disk./.usagePercent‚77.2219142224023$ disk.nvme0c0n1.out‚ 7901.29296875$ memory_percent‚72.61667251586914* proc.memory.percent‚19.910359382629395 cpu‚22.37726732608333 disk.dm-1.in‚0 disk.dm-1.out‚ 13.55859375 disk.sda.out‚ 13.55859375 disk.sda3.out‚ 13.55859375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 885.91015625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 885.91015625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚187913% disk./.usageGB‚1150.6623268127441" disk.nvme0c0n1.in‚ 885.91015625 disk.dm-0.out‚ 7901.29296875 disk.drbd1260.out‚ 0.0546875" disk.nvme0n1.out‚ 7901.29296875' proc.memory.availableMB‚ 4486.484375# proc.memory.rssMB‚ 3262.11328125ˆq&0”+ _wandbruntime‚396  _runtime‚396nlWÛÆ•À ' lr-AdamW/pg2‚0.00013425241746912203 trainer/global_step‚249 % _timestamp‚1.7884701893349288e+09 _runtime‚ 396.090674955 _step‚16 ' lr-AdamW/pg1‚0.00013425241746912203eç~Ê–Ä ' lr-AdamW/pg1‚0.00013425241746912203 ' lr-AdamW/pg2‚0.00013425241746912203 trainer/global_step‚249 % _timestamp‚1.7884701893349288e+09 _runtime‚ 396.090674955 _step‚16'ú0—+ _wandbruntime‚396  _runtime‚396+„ž©Ó'˜ÍO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4580179968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5116695 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39092601270784 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5429371 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 604093117952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11290371 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34872004481024 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37632011421040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3202764 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3716 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6148828 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚249 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5116701 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35708531832832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38235079428604 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4580179968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2227308 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5430077 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3716 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6173670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38231445345500 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5116695 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2226607 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33388756992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38492118046720 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 495178752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33388756992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39092601270784 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39096237556224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3716 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6148828 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 600483224064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3851371008 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 603042579596 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 836403443712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39092601270784 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39096237556224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3716 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6148828 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 600483224064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3851371008 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 603042579596 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 836403443712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3202764 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3716 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6148828 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚249 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5116701 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35708531832832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38235079428604 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4580179968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2227308 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5430077 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3716 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6173670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38231445345500 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5116695 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2226607óÁÙ×0š+ _wandbruntime‚397  _runtime‚397Š0ýÝ&›×M TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 602893705728 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 602893705728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11335388 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 839740121088 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6173395 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 606503612928 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2235466 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 839812812800 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33388756992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35851158885888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3716 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 trainer/global_step‚249 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39252425610240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38645921997312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35011346073088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5137125 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3215569 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72691712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 495178752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6173395 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38384180889922 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚18 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11310514 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3215574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39252425610240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3851371008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33388756992 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚ 397.860594581 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38387814973234 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37782366090144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 601840227746 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4580179968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5451035 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33388756992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 605448883090 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6198263 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35011294870016 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11310514&dDÅá&œÛM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35011294870016 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11310514 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 602893705728 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 602893705728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11335388 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 839740121088 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6173395 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 606503612928 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2235466 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 839812812800 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33388756992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35851158885888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3716 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 trainer/global_step‚249 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39252425610240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38645921997312 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35011346073088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5137125 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3215569 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72691712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 495178752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6173395 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38384180889922 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚18 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11310514 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3215574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39252425610240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3851371008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33388756992 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚ 397.860594581 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38387814973234 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37782366090144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 601840227746 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4580179968 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5451035 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33388756992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 605448883090 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6198263 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877‰÷Ç0+ _wandbruntime‚397  _runtime‚397ú§»nRžÌ 0loss/FBt3/assignment_loss‚0.5175424955316086 0loss/SRqqt2/detection_loss‚5.939867635567983 1loss/SRqqt4/detection_loss‚2.1380119906628834 /loss/FBt4/detection_loss‚13.913032531738281 trainer/global_step‚249 /loss/FRt3/assignment_loss‚7.430257452858819 2loss/SRqqt1/assignment_loss‚3.3975974571557694 /loss/FRt4/detection_loss‚2.7804918839543498 $ _timestamp‚1.788470191108317e+09 .loss/FRt1/assignment_loss‚7.42895095390186 /loss/FRt4/assignment_loss‚7.193005462862411 1loss/FBt1/assignment_loss‚0.09698800377282168 1loss/SRqqt1/detection_loss‚1.6405811019839696 .loss/FBt2/detection_loss‚4.298411608570152 /loss/FBt3/detection_loss‚2.9224025215878737 /loss/FRt2/assignment_loss‚6.923375288645427 _step‚19 2loss/SRqqt3/assignment_loss‚3.7016073390841484 0loss/FBt4/assignment_loss‚0.5218906942754984 .loss/FRt1/detection_loss‚2.609965230812106 0loss/SRqqt3/detection_loss‚5.074484221637249 .loss/FRt2/detection_loss‚7.063577433427175 /loss/FBt1/detection_loss‚10.201162151992321 _runtime‚ 397.865946551 1loss/FBt2/assignment_loss‚0.38842695730353527 .loss/FRt3/detection_loss‚4.318820670247078 &loss/total_loss‚107.24807844833177 epoch‚0 1loss/SRqqt2/assignment_loss‚3.514888392554389 2loss/SRqqt4/assignment_loss‚3.2327389682040497mÖÇVŸÐ 0loss/FBt4/assignment_loss‚0.5218906942754984 .loss/FRt1/detection_loss‚2.609965230812106 0loss/SRqqt3/detection_loss‚5.074484221637249 .loss/FRt2/detection_loss‚7.063577433427175 /loss/FBt1/detection_loss‚10.201162151992321 _runtime‚ 397.865946551 1loss/FBt2/assignment_loss‚0.38842695730353527 .loss/FRt3/detection_loss‚4.318820670247078 &loss/total_loss‚107.24807844833177 epoch‚0 1loss/SRqqt2/assignment_loss‚3.514888392554389 2loss/SRqqt4/assignment_loss‚3.2327389682040497 0loss/FBt3/assignment_loss‚0.5175424955316086 0loss/SRqqt2/detection_loss‚5.939867635567983 1loss/SRqqt4/detection_loss‚2.1380119906628834 /loss/FBt4/detection_loss‚13.913032531738281 trainer/global_step‚249 /loss/FRt3/assignment_loss‚7.430257452858819 2loss/SRqqt1/assignment_loss‚3.3975974571557694 /loss/FRt4/detection_loss‚2.7804918839543498 $ _timestamp‚1.788470191108317e+09 .loss/FRt1/assignment_loss‚7.42895095390186 /loss/FRt4/assignment_loss‚7.193005462862411 1loss/FBt1/assignment_loss‚0.09698800377282168 1loss/SRqqt1/detection_loss‚1.6405811019839696 .loss/FBt2/detection_loss‚4.298411608570152 /loss/FBt3/detection_loss‚2.9224025215878737 /loss/FRt2/assignment_loss‚6.923375288645427 _step‚19 2loss/SRqqt3/assignment_loss‚3.7016073390841484 8Ùúg ‚(:Ü ¶ÇçÔÎÀ¬x proc.cpu.threads‚17 disk.sda3.out‚ 14.15234375 disk.drbd1260.out‚ 0.05859375 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 4486.19140625# proc.memory.rssMB‚ 3262.11328125* proc.memory.percent‚19.910359382629395 cpu‚22.462779451002174 network.recv‚191668) disk./.usagePercent‚77.20940979039987$ disk./.usageGB‚1150.476001739502! disk.nvme0c0n1.in‚ 1001.640625& disk.nvme0c0n1.out‚7952.1162109375! disk.dm-0.out‚7952.1162109375 disk.dm-1.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 14.15234375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 1001.640625$ disk.nvme0n1.out‚7952.1162109375 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚72.6184606552124 network.sent‚387927 disk.dm-0.in‚ 1001.640625 disk.sda.out‚ 14.15234375ì~‘¡‚(:† ¶ÇçÔüÝ™y gpu.0.gpu‚14.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚39.0 gpu.0.powerWatts‚127.171' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Zߨ[¢‚(:Ð ÅÇçÔãÒ”y$ memory_percent‚72.61767387390137* proc.memory.percent‚19.910359382629395 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚8300.7666015625 disk.dm-0.in‚ 1001.640625! disk.dm-0.out‚8300.7666015625 disk.dm-1.in‚0 disk.sda.out‚ 14.5078125 network.sent‚438141 disk.nvme0n1.in‚ 1001.640625 disk.sda1.out‚0 disk.dm-1.out‚14.5625 disk.sda3.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 4486.3203125# proc.memory.rssMB‚ 3262.11328125( disk./.usagePercent‚77.2192957773421! disk.nvme0c0n1.in‚ 1001.640625 disk.sda.in‚0 disk.sda3.out‚14.5625 disk.drbd1260.out‚ 0.05859375 cpu‚22.544685688011 network.recv‚197681% disk./.usageGB‚1150.6233100891113$ disk.nvme0n1.out‚8300.7666015625`Å(:… ÅÇçÔøáòy gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²øT’¤‚(:‡ ÔÇçÔ‡ªx gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.452' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ‹§áE¥‚(:º ÔÇçÔµâ¹y disk.drbd1260.in‚0 network.sent‚443388% disk./.usageGB‚1150.9031562805176 disk.dm-0.in‚ 1001.640625 disk.dm-0.out‚8577.75 disk.dm-1.in‚0 disk.dm-1.out‚ 15.078125 disk.sda3.in‚0 disk.sda.out‚ 15.078125 disk.sda3.out‚ 15.078125 disk.nvme0n1.out‚8577.75 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚72.61762619018555' proc.memory.availableMB‚ 4486.328125 disk.drbd1260.out‚0.0625) disk./.usagePercent‚77.23807649005397 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 1001.640625 cpu‚22.621072272853723 proc.cpu.threads‚17 network.recv‚201429 disk.sda.in‚0 disk.nvme0n1.in‚ 1001.640625 disk.nvme0c0n1.out‚8577.75# proc.memory.rssMB‚ 3262.11328125* proc.memory.percent‚19.910359382629395*ô’¦‚(:‡ ãÇçÔ­ãêw gpu.0.gpu‚83.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.346' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.8692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýüvm_§‚(:Ô ãÇçÔ붰y disk.drbd1260.in‚0# proc.memory.rssMB‚ 3262.11328125 network.recv‚206707" disk.nvme0n1.out‚ 8757.82421875 disk.sda.out‚ 15.44921875 disk.sda3.in‚0 disk.sda3.out‚ 15.44921875* proc.memory.percent‚19.910359382629395 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 15.44921875) proc.memory.availableMB‚ 4486.32421875 cpu‚22.691919264696004 network.sent‚450138) disk./.usagePercent‚77.26537482943753 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 8757.82421875 disk.dm-1.in‚0 disk.drbd1260.out‚0.0625$ memory_percent‚72.61765003204346% disk./.usageGB‚1151.3099212646484 disk.nvme0n1.in‚ 1001.640625! disk.nvme0c0n1.in‚ 1001.640625 disk.dm-0.in‚ 1001.640625 disk.dm-0.out‚ 8757.82421875 disk.sda.in‚0£fuÛ¦¨‚(:’ òÇçÔ·èùw gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.teȶô£÷mp‚43.0 gpu.0.powerWatts‚261.257' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.251400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßâ,ñ]©‚(:Ò òÇçÔÁ¬ïz' disk.nvme0c0n1.out‚9289.54443359375 disk.dm-1.in‚0 disk.sda3.out‚ 15.90234375 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3262.11328125* proc.memory.percent‚19.910359382629395% disk./.usageGB‚1151.3083229064941 disk.sda2.in‚0" disk.dm-0.out‚9289.54443359375 disk.dm-1.out‚ 15.90234375 disk.sda3.in‚0) proc.memory.availableMB‚ 4486.30859375 cpu‚22.760184126728753 network.recv‚210390( disk./.usagePercent‚77.2652675622807 disk.nvme0n1.in‚1024.25% disk.nvme0n1.out‚9289.54443359375 disk.sda1.out‚0 disk.nvme0c0n1.in‚1024.25 disk.sda.in‚0 disk.sda.out‚ 15.90234375 disk.sda1.in‚0 disk.dm-0.in‚1024.25 disk.drbd1260.out‚ 0.0703125# memory_percent‚72.6177453994751 proc.cpu.threads‚17 network.sent‚455316 disk.sda2.out‚0$âFnª‚(:… ÈçÔ›ïËx gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5696 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì#Šrn«‚(:ã ÈçÔ´ÑÎy# proc.memory.rssMB‚ 3262.12890625* proc.memory.percent‚19.910454750061035 cpu‚22.825119007260117% disk./.usageGB‚1151.3083229064941' disk.nvme0c0n1.out‚9348.32177734375 disk.sda.out‚ 16.27734375 disk.sda3.out‚ 16.27734375 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 1123.33203125 disk.sda2.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 1123.33203125 disk.dm-0.in‚ 1123.33203125" disk.dm-0.out‚9348.32177734375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.0703125 disk.sda2.in‚0' proc.memory.availableMB‚ 4485.765625 network.sent‚461944 disk.dm-1.out‚ 16.27734375 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚72.62105941772461 proc.cpu.threads‚17 network.recv‚215701( disk./.usagePercent‚77.2652675622807 disk.dm-1.in‚0% disk.nvme0n1.out‚9348.32177734375/3»|0¬+ _wandbruntime‚481  _runtime‚481Ï2hÆ­À ' lr-AdamW/pg2‚0.00016121073422998991 trainer/global_step‚299 % _timestamp‚1.7884702742562568e+09 _runtime‚ 481.013236587 _step‚20 ' lr-AdamW/pg1‚0.00016121073422998991XYYó樀 ' lr-AdamW/pg1‚0.00016121073422998991 ' lr-AdamW/pg2‚0.00016121073422998991 trainer/global_step‚299 % _timestamp‚1.7884702742562568e+09 _runtime‚ 481.013236587 _step‚20€.0Ç0¯+ _wandbruntime‚481  _runtime‚481!YO³Õ'°ÏO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7405372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚716 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚299 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6137901 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42839289924096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33388756992 % _timestamp‚1.7884702742602665e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6137895 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46902017719808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46905654005248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46181022390272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46180995998720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45868236157808 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2669787 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45148473477840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6137895 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46180995998720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3716 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4580179968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3859759616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46902017719808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7380530 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2670503 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1002714790400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 497327104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33388756992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2184 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2184 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 721021721088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1002644182528 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 _runtime‚ 481.018361268 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 721021721088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6513522 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41836575133696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 719762679968ëïZiÙ'±ÓO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7405372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚716 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚299 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6137901 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42839289924096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33388756992 % _timestamp‚1.7884702742602665e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6137895 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚46902017719808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46905654005248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70607872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46181022390272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚46180995998720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45868236157808 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2669787 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45148473477840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6137895 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46180995998720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3716 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4580179968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3859759616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46902017719808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7380530 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2670503 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1002714790400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 497327104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33388756992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2184 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2184 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 721021721088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1002644182528 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 _runtime‚ 481.018361268 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 721021721088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6513522 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41836575133696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 719762679968R—i–0²+ _wandbruntime‚483  _runtime‚483³H6<Œ$³ÙM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47061842059264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6158325 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3716 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13588290 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1005981640192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6535111 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47058205760512 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6158319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4580179968 _runtime‚ 483.522595912 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47061842059264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3855824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13588290 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 727042109952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41975865522688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 727042109952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46020971702230 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45298802718976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 497327104 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 722168983254 % _timestamp‚1.7884702767648933e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2679287 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 723432202752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46334799949312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1006052234752 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46024605785542 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13563416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2678604 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33388756992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2184 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46334773557760 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6534423 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 Fã&´ÝM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚46020971702230 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45298802718976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 497327104 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 722168983254 % _timestamp‚1.7884702767648933e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2679287 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 723432202752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚46334799949312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1006052234752 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚46024605785542 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13563416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2678604 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33388756992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2184 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚46334773557760 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6534423 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47061842059264 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6158325 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3716 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13588290 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1005981640192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6535111 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47058205760512 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6158319 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4580179968 _runtime‚ 483.522595912 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47061842059264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3855824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13588290 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 727042109952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚41975865522688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 727042109952(¡wS0µ+ _wandbruntime‚483  _runtime‚483äŽDO¶É -loss/FRt1/assignment_loss‚7.3614426352761 /loss/FRt1/detection_loss‚2.7824867877093227 1loss/SRqqt2/detection_loss‚3.9194505153558192 trainer/global_step‚299 2loss/SRqqt2/assignment_loss‚3.5851503641177445 1loss/SRqqt4/assignment_loss‚3.133043758204726 _runtime‚ 483.526465867 /loss/FRt4/assignment_loss‚7.127351923191801 2loss/SRqqt3/assignment_loss‚3.3364458084106445 1loss/SRqqt4/detection_loss‚2.4407931685936255 /loss/FBt1/detection_loss‚10.458433840672175 /loss/FBt3/detection_loss‚1.3952901256885832 .loss/FBt4/detection_loss‚5.777731028470126 0loss/FBt2/assignment_loss‚0.4661645417214933 1loss/SRqqt1/detection_loss‚1.4282096806639122 epoch‚0 /loss/FRt3/assignment_loss‚7.292236341012491 .loss/FRt2/detection_loss‚5.450630249346004 % _timestamp‚1.7884702767707012e+09 0loss/FBt3/assignment_loss‚0.3891962053004704 0loss/FBt4/assignment_loss‚1.0358455382533562 2loss/SRqqt1/assignment_loss‚3.4224202894184685 /loss/FRt2/assignment_loss‚7.461713370154886 0loss/FBt1/assignment_loss‚0.4264095768933354 .loss/FRt3/detection_loss‚4.222115523911811 /loss/FRt4/detection_loss‚3.1075549869144217 0loss/SRqqt3/detection_loss‚4.941270912521606 -loss/FBt2/detection_loss‚6.87215983349344 %loss/total_loss‚97.83354700529637 _step‚23*~HS·Í 0loss/FBt1/assignment_loss‚0.4264095768933354 .loss/FRt3/detection_loss‚4.222115523911811 /loss/FRt4/detection_loss‚3.1075549869144217 0loss/SRqqt3/detection_loss‚4.941270912521606 -loss/FBt2/detection_loss‚6.87215983349344 %loss/total_loss‚97.83354700529637 _step‚23 -loss/FRt1/assignment_loss‚7.3614426352761 /loss/FRt1/detection_loss‚2.7824867877093227 1loss/SRqqt2/detection_loss‚3.9194505153558192 trainer/global_step‚299 2loss/SRqqt2/assignment_loss‚3.5851503641177445 1loss/SRqqt4/assignment_loss‚3.133043758204726 _runtime‚ 483.526465867 /loss/FRt4/assignment_loss‚7.127351923191801 2loss/SRqqt3/assignment_loss‚3.3364458084106445 1loss/SRqqt4/detection_loss‚2.4407931685936255 /loss/FBt1/detection_loss‚10.458433840672175 /loss/FBt3/detection_loss‚1.3952901256885832 .loss/FBt4/detection_loss‚5.777731028470126 0loss/FBt2/assignment_loss‚0.4661645417214933 1loss/SRqqt1/detection_loss‚1.4282096806639122 epoch‚0 /loss/FRt3/assignment_loss‚7.292236341012491 .loss/FRt2/detection_loss‚5.450630249346004 % _timestamp‚1.7884702767707012e+09 0loss/FBt3/assignment_loss‚0.3891962053004704 0loss/FBt4/assignment_loss‚1.0358455382533562 2loss/SRqqt1/assignment_loss‚3.4224202894184685 /loss/FRt2/assignment_loss‚7.461713370154886߬{p¸‚(:å ÈçÔ‰ñ¸| disk.drbd1260.in‚0% disk.nvme0n1.out‚9707.57177734375( proc.memory.availableMB‚ 4486.1953125" proc.memory.rssMB‚ 3262.1328125* proc.memory.percent‚19.910478591918945 network.sent‚467157 network.recv‚219306 disk.sda1.out‚0" disk.dm-0.out‚9707.57177734375 disk.sda3.out‚ 16.92578125! disk.nvme0n1.in‚ 1123.33203125 disk.sda2.out‚0) disk./.usagePercent‚77.26447726702249 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 16.92578125 cpu‚22.886434350107546 proc.cpu.threads‚17 disk.dm-1.out‚ 16.92578125% disk./.usageGB‚1151.2965469360352 disk.sda1.in‚0' disk.nvme0c0n1.out‚9707.57177734375 disk.dm-0.in‚ 1123.33203125 disk.drbd1260.out‚ 0.07421875 disk.sda2.in‚0$ memory_percent‚72.61843681335449# disk.nvme0c0n1.in‚ 1123.33203125 disk.sda3.in‚0¢Ç0™’¹‚(:‡ ÈçÔ¬ë} gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚40.0 gpu.0.powerWatts‚198.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.7976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Ø¯pº‚(:å ŸÈçÔǵx disk.sda2.in‚0) proc.memory.availableMB‚ 4486.17578125* proc.memory.percent‚19.910478591918945 proc.cpu.threads‚17 network.sent‚517690 network.recv‚225636 disk.dm-1.out‚ 17.34765625 disk.sda.out‚ 17.34765625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.07421875% disk.nvme0n1.out‚9822.39599609375$ memory_percent‚72.61855602264404" disk.dm-0.out‚9822.39599609375 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 cpu‚22.944418622655284) disk./.usagePercent‚77.29619736853746' disk.nvme0c0n1.out‚9822.39599609375 disk.dm-0.in‚ 1123.33203125 disk.dm-1.in‚0! disk.nvme0n1.in‚ 1123.33203125 disk.sda1.out‚0" proc.memory.rssMB‚ 3262.1328125$ disk./.usageGB‚1151.769199371338# disk.nvme0c0n1.in‚ 1123.33203125 disk.sda.in‚0 disk.sda3.out‚ 17.34765625 ‡ŽÞœ»‚(:‘ ŸÈçÔä¹°y gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚45.0 gpu.0.powerWatts‚275.604' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.12079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔŒ5oa¼‚(:Ö ®ÈçÔÌÂäx( disk./.usagePercent‚77.2930407956403 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚9978.798828125$ memory_percent‚72.61853218078613( proc.memory.availableMB‚ 4486.1796875* proc.memory.percent‚19.910478591918945 disk.dm-1.out‚ 17.890625 disk.sda3.in‚0! disk.nvme0n1.in‚ 1123.33203125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1123.33203125 disk.dm-0.in‚ 1123.33203125" proc.memory.rssMB‚ 3262.1328125 cpu‚23.000075399414108 network.sent‚523484 network.recv‚230272 disk.sda3.out‚ 17.890625 disk.drbd1260.in‚0# disk.nvme0n1.out‚9978.798828125 disk.dm-0.out‚9978.798828125% disk./.usageGB‚1151.7221641540527 disk.sda.in‚0 disk.sda.out‚ 17.890625 disk.drbd1260.out‚0.078125 disk.sda1.out‚0 proc.cpu.threads‚17I `Õ‘½‚(:† ®ÈçÔ¡õÍy gpu.0.gpu‚33.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚42.0 gpu.0.powerWatts‚176.789' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾TÈ~‘¾‚(:† ½ÈçÔª˜àx gpu.0.gpu‚27.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.436' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ‰e¿‚(:Ú ½ÈçÔ¨ªõz disk.dm-0.in‚ 1264.6171875 disk.dm-1.out‚ 18.2578125 disk.sda.in‚0 disk.drbd1260.in‚0$ disk.nvme0n1.out‚10264.486328125 disk.sda1.in‚0) proc.memory.availableMB‚ 4487.54296875% disk./.usageGB‚1151.7925491333008! disk.dm-0.out‚10264.486328125 cpu‚23.05415213335181 proc.cpu.threads‚17 disk.sda3.in‚0 network.sent‚531683 network.recv‚235217) disk./.usagePercent‚77.29776439065455 disk.sda.out‚ 18.2578125 disk.nvme0n1.in‚ 1264.6171875 disk.sda2.out‚0" proc.memory.rssMB‚ 3262.1328125& disk.nvme0c0n1.out‚10264.486328125 disk.dm-1.in‚0 disk.sda3.out‚ 18.2578125 disk.drbd1260.out‚0.078125 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚72.61021137237549* proc.memory.percent‚19.910478591918945" disk.nvme0c0n1.in‚ 1264.6171875m©¹aÀ‚(:’ ÌÈçÔ¥Ëw gpu.0.gpu‚92.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51423645019531* gpu.0.memoryAllocatedBytes‚ 33942470656 gpu.0.temp‚43.0 gpu.0.powerWatts‚251.976' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.395199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ) ¦iÁ‚(:Þ ÌÈçÔÚ”x) proc.memory.availableMB‚ 4487.54296875* proc.memory.percent‚19.910478591918945 disk.sda.out‚ 18.62890625 disk.nvme0n1.in‚ 1264.6171875 disk.dm-1.in‚0 disk.dm-1.out‚ 18.62890625" proc.memory.rssMB‚ 3262.1328125 proc.cpu.threads‚17) disk./.usagePercent‚77.29776567069221$ disk./.usageGB‚1151.792568206787 disk.sda.in‚0 disk.sda3.out‚ 18.62890625 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1264.6171875 cpu‚23.10417775655641 network.recv‚238828 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.08203125$ disk.nvme0n1.out‚10613.095703125 disk.sda2.in‚0& disk.nvme0c0n1.out‚10613.095703125 network.sent‚534909 disk.sda3.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 1264.6171875! disk.dm-0.out‚10613.095703125$ memory_percent‚72.61021137237549L>&0Â+ _wandbruntime‚566  _runtime‚566„–6{Äþ & lr-AdamW/pg1‚0.0001881690509908578 & lr-AdamW/pg2‚0.0001881690509908578 trainer/global_step‚349 % _timestamp‚1.7884703596333375e+09 _runtime‚ 566.389253856 _step‚24Ö›)ÈÄ & lr-AdamW/pg1‚0.0001881690509908578 & lr-AdamW/pg2‚0.0001881690509908578 trainer/global_step‚349 % _timestamp‚1.7884703596333375e+09 _runtime‚ 566.389253856 _step‚246 ã0Å+ _wandbruntime‚566  _runtime‚566aúŠµÔ'ÆÎO ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 843700090612 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2184 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7159095 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53508661053220 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 845170112000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33388756992 % _timestamp‚1.7884703596342793e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7595663 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52664935534640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15771327 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 841560218112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3716 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8612232 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7159095 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 845170112000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3716 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4483264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2184 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7159101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7159101 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48801043380224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1168949788672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7596368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4580179968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52664960962608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53869873950720 trainer/global_step‚349 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54715070454272 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4483269 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33388756992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4580179968 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54711434168832 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8637074 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33388756992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4580179968 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54711434168832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3112399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49970117076992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 841560218112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3716 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 K?DeviceStatsMonitor.on_train_batch_start/active.smalkNp/l_pool.freed‚8612232 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7159095 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 845170112000 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3716 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4483264 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2184 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7159101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7159101 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48801043380224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1168949788672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7596368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4580179968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52664960962608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53869873950720 trainer/global_step‚349 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54715070454272 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4483269 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33388756992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4580179968 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54711434168832 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8637074 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33388756992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4580179968 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54711434168832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3112399 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49970117076992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 568.211417564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8661667 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48940384972288 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 846106394106 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7179519 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52815290203744 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15841192 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 847580606976 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7179525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3121882 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15841192 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 843970699776 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4580179968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3716 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4496069 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54867622209536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3121187 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33388756992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15816318 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3716 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2184 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72691712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2184 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7617256 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7617956 _step‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50112612326912 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 842497738762 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50112736221696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33388756992 trainer/global_step‚349 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4580179968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4580179968ÝHË7ã&ÊÝM I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8636799 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72691712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2184 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7617256 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7617956 _step‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50112612326912 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 842497738762 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50112736221696 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33388756992 trainer/global_step‚349 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4580179968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4580179968 % _timestamp‚1.7884703614504664e+09 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 568.211417564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8661667 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48940384972288 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 846106394106 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7179519 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52815290203744 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15841192 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 847580606976 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7179525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3121882 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15841192 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 843970699776 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4580179968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3716 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚700 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4496069 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54867622209536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3121187 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33388756992 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15816318 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3716 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2184 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1335202934784 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 965708609024 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8180295 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8180295 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9868776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18024229 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60181423019408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62520850617856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 962098715136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8678770 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62520850617856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3555256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 _runtime‚ 640.371142594 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9843934 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 962098715136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57100765764608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5123519 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61558778294272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9868776 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62524486903296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 % _timestamp‚1.7884704336152298e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 _step‚29 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8180301 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55765562829824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61558751902720 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62524486903296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 964028846120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 trainer/global_step‚399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60181397591440§®]µÙ'ÝÓO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9868776 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62524486903296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 % _timestamp‚1.7884704336152298e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 _step‚29 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8180301 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55765562829824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61558751902720 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62524486903296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 964028846120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 trainer/global_step‚399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60181397591440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18049077 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61141817782424 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1335202934784 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 965708609024 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8180295 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8180295 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9868776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18024229 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60181423019408 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62520850617856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 962098715136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8678770 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62520850617856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3555256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 _runtime‚ 640.371142594 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9843934 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 962098715136 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57100765764608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5123519 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61558778294272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024zÝrÑ0Þ+ _wandbruntime‚641  _runtime‚64194ÞßØM BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 964509196800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18094094 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8700478 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61712529461760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62680674957312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 _step‚30 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9893369 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18094094 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9868501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55904904421888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9893369 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3564159 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788470434658831e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18069220 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 968119104000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18069220 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9868501 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 966435149614 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62677038658560 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8200725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3564831 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 641.415649919 [HDeviceStatsMonitor.on_train_batch_end/i/muaO nactive_split_bytes.all.allocated‚57243506544128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72691712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62680674957312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61712555853312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1338529430528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61294553326846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5136324 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61298187410158 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 trainer/global_step‚399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8701155 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60331752260544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61712529461760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8200719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1338602122240…бúâ&àÜM TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 964509196800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 964509196800 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18094094 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8700478 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61712529461760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62680674957312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 _step‚30 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9893369 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18094094 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9868501 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55904904421888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9893369 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3564159 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788470434658831e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18069220 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 968119104000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18069220 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9868501 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 966435149614 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62677038658560 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8200725 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3564831 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 641.415649919 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57243506544128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72691712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62680674957312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61712555853312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1338529430528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61294553326846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5136324 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61298187410158 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 trainer/global_step‚399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8701155 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60331752260544 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61712529461760 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8200719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1338602122240 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67697885076208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1082637212160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69247656246272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69247629854720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 _step‚33 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚716 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64231654107136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚449 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 _runtime‚ 705.41028583 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1086247106048 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67697859648240 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70330267066880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70333903352320 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9201501 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68782242677836 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1080748946492 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20277131 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9201501 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9201495 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 $ _timestamp‚1.788470498654317e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69247629854720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1501520624640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4000015 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11100478 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9763784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11075636 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20301979 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9201495 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70333903352320 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70330267066880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1082637212160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9763063 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1086247106048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1501447919616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3999299 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68778608594732 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64231530199040p¼ìÜ­óÛO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11075636 TGDeviceStatsMhs-•4'onitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20301979 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9201495 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70333903352320 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70330267066880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1082637212160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9763063 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1086247106048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1501447919616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3999299 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68778608594732 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64231530199040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62730133482496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67697885076208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1082637212160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69247656246272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69247629854720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 _step‚33 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚716 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64231654107136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚449 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 _runtime‚ 705.41028583 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1086247106048 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67697859648240 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70330267066880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚70333903352320 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9201501 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68782242677836 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1080748946492 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20277131 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9201501 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9201495 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 $ _timestamp‚1.788470498654317e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69247629854720 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1501520624640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4000015 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11100478 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9763784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0!,"Ü0ô+ _wandbruntime‚706  _runtime‚706Nìè&õâM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69401407413760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11100203 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1088657601024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69401433805312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67848214317344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1088657601024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9785441 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1504849224192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5776574 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5776569 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68931344139154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9784760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20322122 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62869423871488 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68934978222466 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9221919 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20346996 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70490091406336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4008191 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1085047693824 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9221925 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 _step‚34‚üEì&öæM ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70490091406336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4008191 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1085047693824 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9221925 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 _step‚34 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69401407413760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11100203 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1088657601024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69401433805312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67848214317344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1088657601024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9785441 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1504849224192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5776574 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5776569 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68931344139154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9784760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20322122 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62869423871488 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68934978222466 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9221919 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20346996 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921"ƒ ©g0÷+ _wandbruntime‚706  _runtime‚706ÌèP¦OøÉ 1loss/FBt1/assignment_loss‚0.44859618865503464 .loss/FBt1/detection_loss‚9.536073521925854 trainer/global_step‚449 /loss/FRt1/assignment_loss‚7.343862771987915 /loss/FRt2/assignment_loss‚7.126425853142371 1loss/SRqqt1/detection_loss‚1.5038690122239518 _step‚35 2loss/SRqqt2/assignment_loss‚3.4248324973242625 .loss/FRt2/detection_loss‚2.920718248073871 .loss/FBt2/detection_loss‚4.546517074108124 1loss/FBt3/assignment_loss‚0.22179051034302855 /loss/FRt3/detection_loss‚2.6351387346429482 0loss/SRqqt3/detection_loss‚2.554450824304863 0loss/FRt4/assignment_loss‚7.2190641580626025 1loss/SRqqt1/assignment_loss‚2.164297917876581 1loss/SRqqt3/assignment_loss‚3.182514711662575 ,loss/FBt3/detection_loss‚4.9916478206714 /loss/FBt4/detection_loss‚0.7039255958852645 epoch‚0 1loss/SRqqt4/assignment_loss‚4.456738415218535 0loss/FBt4/assignment_loss‚0.6925571288986909 /loss/FRt4/detection_loss‚2.7761425745110353 0loss/SRqqt2/detection_loss‚4.285355733973639 0loss/SRqqt4/detection_loss‚6.125627204066231 % _timestamp‚1.7884704999221222e+09 /loss/FRt3/assignment_loss‚7.428388186863491 0loss/FBt2/assignment_loss‚0.2528937262579193 /loss/FRt1/detection_loss‚2.6106960490859787 %loss/total_loss‚89.15212445976617 _runtime‚ 706.677571287º$ùÍ _runtime‚ 706.677571287 1loss/FBt1/assignment_loss‚0.44859618865503464 .loss/FBt1/detection_loss‚9.536073521925854 trainer/global_step‚449 /loss/FRt1/assignment_loss‚7.343862771987915 /loss/FRt2/assignment_loss‚7.126425853142371 1loss/SRqqt1/detection_loss‚1.5038690122239518 _step‚35 2loss/SRqqt2/assignment_loss‚3.4248324973242625 .loss/FRt2/detection_loss‚2.920718248073871 .loss/FBt2/detection_loss‚4.546517074108124 1loss/FBt3/assignment_loss‚0.22179051034302855 /loss/FRt3/detection_loss‚2.6351387346429482 0loss/SRqqt3/detection_loss‚2.554450824304863 0loss/FRt4/assignment_loss‚7.2190641580626025 1loss/SRqqt1/assignment_loss‚2.164297917876581 1loss/SRqqt3/assignment_loss‚3.182514711662575 ,loss/FBt3/detection_loss‚4.9916478206714 /loss/FBt4/detection_loss‚0.7039255958852645 epoch‚0 1loss/SRqqt4/assignment_loss‚4.456738415218535 0loss/FBt4/assignment_loss‚0.6925571288986909 /loss/FRt4/detection_loss‚2.7761425745110353 0loss/SRqqt2/detection_loss‚4.285355733973639 0loss/SRqqt4/detection_loss‚6.125627204066231 % _timestamp‚1.7884704999221222e+09 /loss/FRt3/assignment_loss‚7.428388186863491 0loss/FBt2/assignment_loss‚0.2528937262579193 /loss/FRt1/detection_loss‚2. ¿Ðž;6106960490859787 %loss/total_loss‚89.15212445976617#&gºìiú‚(:Þ ñÉçÔµ²y disk.sda.in‚0 network.recv‚287885) disk./.usagePercent‚77.29744822134977& disk.nvme0n1.out‚14298.35498046875( disk.nvme0c0n1.out‚14298.35498046875 disk.dm-1.out‚ 23.6953125 disk.sda.out‚ 23.6953125) proc.memory.availableMB‚ 4487.01953125! proc.memory.rssMB‚ 3262.140625 network.sent‚685344 disk.drbd1260.out‚ 0.1015625 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1684.453125* proc.memory.percent‚19.910526275634766 proc.cpu.threads‚17 disk.sda3.out‚ 23.6953125 disk.nvme0n1.in‚ 1684.453125 disk.sda2.out‚0$ memory_percent‚72.61340618133545 cpu‚23.526022228632375% disk./.usageGB‚1151.7878379821777 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 1684.453125# disk.dm-0.out‚14298.35498046875 disk.dm-1.in‚0Dà<ë’û‚(:‡ ñÉçÔ§ú¬y gpu.0.gpu‚52.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚39.0 gpu.0.powerWatts‚116.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª~S-ü‚(:„ €ÊçÔ¯Ùy gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚238.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Briký‚(:à €ÊçÔ›š×| cpu‚23.557246572168623 network.recv‚292366! disk.nvme0c0n1.in‚ 1684.453125 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚77.29744822134977% disk./.usageGB‚1151.7878379821777( disk.nvme0c0n1.out‚14779.00146484375# disk.dm-0.out‚14779.00146484375 disk.dm-1.in‚0 disk.sda3.out‚ 24.17578125 disk.drbd1260.out‚ 0.10546875& disk.nvme0n1.out‚14779.00146484375 network.sent‚735823 disk.dm-1.out‚ 24.17578125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚72.61343002319336! proc.memory.rssMB‚ 3262.140625 disk.dm-0.in‚ 1684.453125 disk.sda.out‚ 24.17578125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 1684.453125' proc.memory.availableMB‚ 4487.015625* proc.memory.percent‚19.910526275634766o4 kþ‚(:à ÊçÔµ¸¸y$ memory_percent‚72.61343002319336 cpu‚23.585375511345166 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚14929.39599609375 disk.sda2.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 3262.140625 network.sent‚740301! disk.nvme0c0n1.in‚ 1684.453125( disk.nvme0c0n1.out‚14929.39599609375 disk.sda.out‚ 24.80859375 disk.sda3.out‚ 24.80859375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1684.453125# disk.dm-0.out‚14929.39599609375 disk.dm-1.out‚ 24.80859375 disk.drbd1260.out‚ 0.10546875' proc.memory.availableMB‚ 4487.015625* proc.memory.percent‚19.910526275634766 network.recv‚297370) disk./.usagePercent‚77.29776874278264% disk./.usageGB‚1151.7926139831543 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 1684.453125 disk.sda2.in‚0®Y–œÿ‚(:‘ ÊçÔñŸÓy gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚186.997' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚37.39940000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 «J,k€‚(:à žÊçÔ›ŠÊx disk.sda2.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 1684.453125) proc.memory.availableMB‚ 4487.01171875) disk./.usagePercent‚77.29777386293331 disk.sda.out‚ 25.15234375 disk.sda3.out‚ 25.15234375 disk.drbd1260.in‚0& disk.nvme0n1.out‚15025.09228515625 cpu‚23.614133536127262 proc.cpu.threads‚17 network.recv‚300847 disk.dm-0.in‚ 1684.453125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 1684.453125 disk.sda1.in‚0 network.sent‚745257% disk./.usageGB‚1151.7926902770996 disk.dm-1.out‚ 25.15234375( disk.nvme0c0n1.out‚15025.09228515625$ memory_percent‚72.61345386505127! proc.memory.rssMB‚ 3262.140625* proc.memory.percent‚19.910526275634766# disk.dm-0.out‚15025.09228515625 disk.sda.in‚0 disk.drbd1260.out‚0.109375 disk.sda2.in‚0Ù80Ò›‚(: žÊçÔØÇãy gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚39.0 gpu.0.powerWatts‚201.735' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚40.34700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òús‚‚(:’ ­ÊçÔýÏóx gpu.0.gpu‚92.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚250.918' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.183600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö‘lòiƒ‚(:Þ ­ÊçÔÒÚ¹y disk.dm-1.out‚ 25.5546875 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 1815.0234375 disk.sda2.out‚0) proc.memory.availableMB‚ 4487.07421875 network.recv‚305631) disk./.usagePercent‚77.30196291421609( disk.nvme0c0n1.out‚15480.47119140625 disk.dm-0.in‚ 1815.0234375 disk.dm-1.in‚0& disk.nvme0n1.out‚15480.47119140625 disk.sda2.in‚0 disk.sda1.in‚0# disk.dm-0.out‚15480.47119140625 disk.sda3.out‚ 25.5546875 disk.drbd1260.out‚0.109375 disk.sda1.out‚0# memory_percent‚72.6130723953247! proc.memory.rssMB‚ 3262.140625* proc.memory.percent‚19.910526275634766 cpu‚23.641523102164406 network.sent‚751372$ disk./.usageGB‚1151.855110168457" disk.nvme0c0n1.in‚ 1815.0234375 disk.sda.out‚ 25.5546875 proc.cpu.threads‚17Ôë¹0„+ _wandbruntime‚788  _runtime‚788êÌŸÄ…¾ ' lr-AdamW/pg1‚0.00026904400127346145 ' lr-AdamW/pg2‚0.00026904400127346145 trainer/global_step‚499 # _timestamp‚1.78847058134458e+09 _runtime‚ 788.101223329 _step‚36‡L8Ȇ ' lr-AdamW/pg1‚0.00026904400127346145 ' lr-AdamW/pg2‚0.00026904400127346145 trainer/global_step‚499 # _timestamp‚1.78847058134458e+09 _runtime‚ 788.101223329 _step‚36${ö›0‡+ _wandbruntime‚788  _runtime‚788{CÁIå'ˆßO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10222701 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78139683515904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76936534198272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1201077702000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10848014 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4443293 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1206785603072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1206785603072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884705813482049e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78143319801344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12307338 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75214347133008 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76936507806720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76419033490144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1203175709184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69694601729024 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78139683515904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71362898354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1203175709184 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22554881 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10222701 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12332180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10222695 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 788.105328048 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12332180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6404019 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78143319801344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76936534198272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 _step‚37 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952ƒõÃ}é'‰ãO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4443995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12332180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10222695 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 788.105328048 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12332180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6404019 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78143319801344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76936534198272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 _step‚37 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10222701 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78139683515904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76936534198272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1201077702000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10848014 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4443293 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1206785603072 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1206785603072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884705813482049e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78143319801344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12307338 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75214347133008 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76936507806720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76419033490144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1203175709184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69694601729024 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78139683515904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71362898354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1203175709184 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22554881 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10222701 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6416819 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1203484005286 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _step‚38 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77090311757312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22599898 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6416824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10243119 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1209196098048 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78299507855360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22599898 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77090311757312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10869601 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75364650946176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 trainer/global_step‚499 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10868902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76571769034774 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22575024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71505407537664 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6416819 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1203484005286 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _step‚38 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77090311757312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22599898 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6416824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10243119 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1209196098048 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78299507855360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22599898 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77090311757312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10869601 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75364650946176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 trainer/global_step‚499 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10868902 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76571769034774 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22575024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71505407537664 Bã,û‘’‚(:† ËÊçÔÿŸˆy gpu.0.gpu‚77.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚258.575' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06zša“‚(:Ö ËÊçÔÑ·ïy disk.sda1.in‚0( proc.memory.availableMB‚ 4486.9921875 cpu‚23.691169418024018 proc.cpu.threads‚17 network.recv‚314830$ disk./.usageGB‚1150.955379486084 disk.dm-0.in‚ 1845.28125 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚72.61357307434082! proc.memory.rssMB‚ 3262.140625 disk.nvme0c0n1.in‚ 1845.28125' disk.nvme0c0n1.out‚16409.3056640625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 1845.28125* proc.memory.percent‚19.910526275634766( disk./.usagePercent‚77.2415812331977" disk.dm-0.out‚16409.3056640625 disk.sda.out‚ 26.7421875 disk.sda3.in‚0 disk.sda3.out‚ 26.7421875 disk.drbd1260.in‚0% disk.nvme0n1.out‚16409.3056640625 disk.sda2.in‚0 network.sent‚808236 disk.dm-1.out‚ 26.7421875 disk.drbd1260.out‚ 0.11328125 disk.sda2.out‚0ÙQM­f”‚(:Û ÚÊçÔƒú‡y disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.11328125$ disk.nvme0n1.out‚16707.712890625 disk.sda1.in‚0) proc.memory.availableMB‚ 4486.98828125* proc.memory.percent‚19.910550117492676 cpu‚23.714156707838548 disk.nvme0c0n1.in‚ 1845.28125& disk.nvme0c0n1.out‚16707.712890625! disk.dm-0.out‚16707.712890625 disk.sda.out‚ 27.16015625 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3262.14453125) disk./.usagePercent‚77.25836022299936 disk.sda3.out‚ 27.16015625 disk.nvme0n1.in‚ 1845.28125 disk.sda2.out‚0$ memory_percent‚72.61359691619873 network.sent‚811224 network.recv‚318232% disk./.usageGB‚1151.2053985595703 disk.dm-0.in‚ 1845.28125 disk.dm-1.out‚ 27.16015625 disk.sda.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0k‰Ø•‚(:’ ÚÊçÔÈáy gpu.0.gpu‚54.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚216.685' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.336999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„àC¯’–‚(:‡ éÊçÔäÑ­y gpu.0.gpu‚45.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AËz8d—‚(:Ù éÊçÔýúy disk.sda1.out‚0 disk.nvme0c0n1.in‚ 1845.28125 disk.sda.out‚ 27.50390625( proc.memory.availableMB‚ 4486.9765625# proc.memory.rssMB‚ 3262.14453125 disk.sda1.in‚0& disk.nvme0c0n1.out‚16992.310546875 disk.dm-1.out‚ 27.50390625 disk.sda.in‚0 cpu‚23.736048233132443 proc.cpu.threads‚17 network.sent‚817522 network.recv‚322934) disk./.usagePercent‚77.25678552065625 disk.sda2.in‚0 disk.dm-0.in‚ 1845.28125! disk.dm-0.out‚16992.310546875 disk.sda3.in‚0* proc.memory.percent‚19.910550117492676$ disk.nvme0n1.out‚16992.310546875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 27.50390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1171875$ memory_percent‚72.61366844177246% disk./.usageGB‚1151.1819343566895 disk.nvme0n1.in‚ 1845.28125ZŒ¶m˜‚(:â øÊçÔ“Ž¶x& disk.nvme0n1.out‚17296.22607421875 disk.sda2.out‚0* proc.memory.percent‚19.910550117492676 network.sent‚822339! disk.nvme0c0n1.in‚ 1963.828125 disk.dm-1.in‚0# proc.memory.rssMB‚ 3262.14453125 proc.cpu.threads‚17 network.recv‚326629# disk.dm-0.out‚17296.22607421875 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 1963.828125 disk.sda2.in‚0 disk.sda1.out‚0% disk./.usageGB‚1151.7119178771973 disk.dm-1.out‚ 27.83203125 disk.sda.out‚ 27.83203125 disk.sda1.in‚0# memory_percent‚72.6121187210083) proc.memory.availableMB‚ 4487.23046875 cpu‚23.757378364570748) disk./.usagePercent‚77.29235315940332( disk.nvme0c0n1.out‚17296.22607421875 disk.dm-0.in‚ 1963.828125 disk.sda3.out‚ 27.83203125 disk.drbd1260.out‚ 0.1171875î.’™‚(:‡ øÊçÔ…ïÚx gpu.0.gpu‚53.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚196.054' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.2108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@6¯©0š+ _wandbruntime‚869  _runtime‚869¤ªpoÛ½ & lr-AdamW/pg1‚0.0002960023180343293 & lr-AdamW/pg2‚0.0002960023180343293 trainer/global_step‚549 $ _timestamp‚1.788470662287521e+09 _runtime‚ 869.043082319 _step‚40–”Å^ÇœÁ & lr-AdamW/pg1‚0.0002960023180343293 & lr-AdamW/pg2‚0.0002960023180343293 trainer/global_step‚549 $ _timestamp‚1.788470662287521e+09 _runtime‚ 869.043082319 _step‚40(:±l0+ _wandbruntime‚869  _runtime‚869¾Z÷G žÝO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24782935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7044264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76659172381696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84625412150272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76659121178624 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13563882 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13539040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70607872 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85952736250368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1834653796864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85949099964928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1834724404736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85952736250368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1327324100096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84052190219348 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82730809189808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4887096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11243895 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11930646 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78493896786432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11243901 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13539040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82730783761840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1325015112644 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _step‚41 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78493774975488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 trainer/global_step‚549 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84625385758720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1323714206208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1323714206208 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84055824302452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚719 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84625412150272 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4886382 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitorÎ-¹œ.on_train_batch_start/segment.small_pool.current‚2185 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84625385758720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24807783 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11931365 _runtime‚ 869.044412908 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85949099964928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11243901 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13563882 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1321406457508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 # _timestamp‚1.78847066228871e+09¸ñë]ç'ŸáO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4887096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11243895 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11930646 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78493896786432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11243901 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13539040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82730783761840 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1325015112644 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _step‚41 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78493774975488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 trainer/global_step‚549 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84625385758720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1323714206208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1323714206208 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84055824302452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚719 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84625412150272 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4886382 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84625385758720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24807783 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11931365 _runtime‚ 869.044412908 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85949099964928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11243901 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13563882 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1321406457508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 # _timestamp‚1.78847066228871e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24782935 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7044264 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76659172381696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84625412150272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76659121178624 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13563882 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13539040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70607872 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85952736250368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1834653796864 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85949099964928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1834724404736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85952736250368 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1327324100096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84052190219348 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82730809189808)iœ1œ ‚(:‘ ‡ËçÔ’ÜÕy gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚46.0 gpu.0.powerWatts‚266.076' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.21520000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'>gl¡‚(:á ‡ËçԪϑz disk.sda1.in‚0) disk./.usagePercent‚77.29065966956432% disk./.usageGB‚1151.6866836547852 disk.sda3.in‚0 disk.nvme0n1.in‚ 1963.828125 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 1963.828125# disk.dm-0.out‚17605.89794921875 disk.sda3.out‚ 28.51171875& disk.nvme0n1.out‚17605.89794921875( disk.nvme0c0n1.out‚17605.89794921875 disk.dm-1.out‚ 28.51171875 disk.sda.in‚0 disk.sda.out‚ 28.51171875$ memory_percent‚72.61514663696289* proc.memory.percent‚19.910573959350586 disk.sda1.out‚0 network.recv‚331492 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.12109375 cpu‚23.778138019779107 network.sent‚828680 disk.sda2.out‚0 disk.dm-0.in‚ 1963.828125 disk.dm-1.in‚0' proc.memory.availableMB‚ 4486.734375" proc.memory.rssMB‚ 3262.1484375 proc.cpu.threads‚17«µì0¢+ _wandbruntime‚870  _runtime‚870ÛU5=ï&£éM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84779163317760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24827926 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 % _timestamp‚1.7884706633198187e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4895845 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78636373379584 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84204925763770 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11264325 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4895170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1837961811968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84779189709312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24852800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24852800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1838032406528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11952919 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11952239 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1327421416138 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11264325 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76798462770688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84208559847082 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11264319 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13588475 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82881138430944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 _runtime‚ 870.076251589 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76798411567616 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 trainer/global_step‚549 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86108924304384 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82881113002976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24827926 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7057074 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7057069 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚42 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86108924304384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160õ÷]w%$¤íM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84779163317760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1329734595072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1323812760794 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82881113002976 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24827926 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7057074 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7057069 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚42 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86108924304384 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84779163317760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24827926 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 % _timestamp‚1.7884706633198187e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4895845 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78636373379584 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84204925763770 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11264325 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4895170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1837961811968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84779189709312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24852800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24852800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1838032406528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11952919 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11952239 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1327421416138 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11264325 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76798462770688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84208559847082 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11264319 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13588475 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82881138430944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 @:DeviceStatsMonitor.on_train_batch_end/num_syhR¾€Înc_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 _runtime‚ 870.076251589 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76798411567616 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 trainer/global_step‚549 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86108924304384 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93762152699392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27035837 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1444252703232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5329000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1441735213016 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93758516413952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92314290102272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91688981031656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12265095 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1444252703232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14795584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90247271246608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7684514 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93762152699392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 957.138710492 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92314290102272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1447862597120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7684519 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90247245818640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1447862597120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12265101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14795584 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85624524730368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2000884102144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚45 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93758516413952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92314263710720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13013519 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144Zuëâè'·âO _step‚45 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93758516413952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92314263710720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13013519 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14770742 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚92314263710720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14770742 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85624648638464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚599 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93762152699392 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27035837 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1444252703232 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5329000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1441735213016 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93758516413952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92314290102272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91688981031656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12265095 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1444252703232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14795584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90247271246608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7684514 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚93762152699392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 957.138710492 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92314290102272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1447862597120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7684519 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90247245818640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1447862597120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12265101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14795584 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85624524730368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2000884102144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0-…Š#0¸+ _wandbruntime‚959  _runtime‚959ÚÙt̹éM MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90397575059776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 trainer/global_step‚599 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93918340753408 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27105702 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7697324 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5336991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83762982220288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch}Ò«g# _end/segment.all.current‚3717 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚72691712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12285519 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92468041269760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13035013 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92468067661312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1450273092096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884707528099773e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27080828 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93914704454656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13034310 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5337689 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12285525 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27105702 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1446663184896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92468041269760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27080828 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83762931017216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2004296222720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1447750171646 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1450273092096 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93914704454656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13034310 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5337689 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12285525 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27105702 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1446663184896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92468041269760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27080828 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83762931017216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2004296222720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1447750171646 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1450273092096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90397575059776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 trainer/global_step‚599 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93918340753408 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27105702 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7697324 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5336991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83762982220288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185.¯˜˜0»+ _wandbruntime‚959  _runtime‚9599ÐÚU¼Ï .loss/FBt3/detection_loss‚3.870180959728631 trainer/global_step‚599 1loss/FBt4/assignment_loss‚0.27392197814224345 /loss/FRt4/assignment_loss‚6.649263818213281 0loss/FBt1/assignment_loss‚0.3571612630105512 epoch‚0 0loss/FBt3/assignment_loss‚0.6126105210048329 /loss/FRt4/detection_loss‚2.6131782430283566 /loss/FBt2/detection_loss‚1.8325665138948422 /loss/FRt3/assignment_loss‚7.207111318906148 2loss/SRqqt4/assignment_loss‚3.6173472881317137 /loss/FRt3/detection_loss‚3.4223356942335763 0loss/SRqqt3/detection_loss‚2.663275306639464 2loss/SRqqt3/assignment_loss‚2.8882986942063207 /loss/FRt1/detection_loss‚1.9470204815268517 0loss/SRqqt1/detection_loss‚0.753318114128042 /loss/FBt1/detection_loss‚3.0495971174616563 1loss/FBt2/assignment_loss‚0.38511316569660115 /loss/FBt4/detection_loss‚3.5182273705800373 % _timestamp‚1.7884707528171983e+09 _runtime‚ 959.573919071 1loss/SRqqt2/detection_loss‚3.1023414421964577 %loss/total_loss‚72.84168490429505 _step‚47 .loss/FRt1/assignment_loss‚5.29900856912136 2loss/SRqqt1/assignment_loss‚2.5948421909258914 1loss/SRqqt2/assignment_loss‚3.558175908194648 1loss/SRqqt4/detection_loss‚3.0517881321907043 /loss/FRt2/assignment_loss‚6.653250016664204 /loss/FRt2/detection_loss‚2.9217507964686344îuý`Y½Ó /loss/FRt2/assignment_loss‚6.653250016664204 /loss/FRt2/detection_loss‚2.9217507964686344 .loss/FBt3/detection_loss‚3.870180959728631 trainer/global_step‚599 1loss/FBt4/assignment_loss‚0.27392197814224345 /loss/FRt4/assignment_loss‚6.649263818213281 0loss/FBt1/assignment_loss‚0.3571612630105512 epoch‚0 0loss/FBt3/assignment_loss‚0.6126105210048329 /loss/FRt4/detection_loss‚2.6131782430283566 /loss/FBt2/detection_loss‚1.8325665138948422 /loss/FRt3/assignment_loss‚7.207111318906148 2loss/SRqqt4/assignment_loss‚3.6173472881317137 /loss/FRt3/detection_loss‚3.4223356942335763 0loss/SRqqt3/detection_loss‚2.663275306639464 2loss/SRqqt3/assignment_loss‚2.8882986942063207 /loss/FRt1/detection_loss‚1.9470204815268517 0loss/SRqqt1/detection_loss‚0.753318114128042 /loss/FBt1/detection_loss‚3.0495971174616563 1loss/FBt2/assignment_loss‚0.38511316569660115 /loss/FBt4/detection_loss‚3.5182273705800373 % _timestamp‚1.7884707528171983e+09 _runtime‚ 959.573919071 1loss/SRqqt2/detection_loss‚3.1023414421964577 %loss/total_loss‚72.84168490429505 _step‚47 .loss/FRt1/assignment_loss‚5.29900856912136 2loss/SRqqt1/assignment_loss‚2.5948421909258914 1loss/SRqqt2/assignment_loss‚3.558175908194648 1loss/SRqqt4/detection_loss‚3.0517881321907043/G?ŒÔ¾‚(:„ áËçÔèúów gpu.0.gpu‚35.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.84' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{¸ÅSq¿‚(:æ áËçÔ¹°Ì{) proc.memory.availableMB‚ 4486.33203125% disk./.usageGB‚1151.5419235229492 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 31.26953125 disk.sda.in‚0 disk.sda3.out‚ 31.26953125 network.recv‚357630) disk./.usagePercent‚77.28094469565497 disk.sda1.in‚0 disk.dm-0.in‚ 2122.8671875# disk.dm-0.out‚19540.43115234375 disk.sda.out‚ 31.26953125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1328125 network.sent‚927559 disk.nvme0n1.in‚ 2122.8671875& disk.nvme0n1.out‚19540.43115234375$ memory_percent‚72.61760234832764# proc.memory.rssMB‚ 3262.15234375* proc.memory.percent‚19.910597801208496 cpu‚23.890723425611796 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 2122.8671875( disk.nvme0c0n1.out‚19540.43115234375 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0‡¡’…À‚(:’ ðËçÔ¨ëÓx gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚246.593' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.318599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a€«pÁ‚(:å ðËçÔÀô†z cpu‚23.907493226004497 proc.cpu.threads‚17 network.sent‚976237 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 2244.94140625" disk.dm-0.out‚19921.3076171875 disk.dm-1.out‚ 31.59765625 disk.sda3.out‚ 31.59765625$ memory_percent‚72.61614799499512( proc.memory.availableMB‚ 4486.5703125 network.recv‚362265) disk./.usagePercent‚77.28093087124812 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1328125! disk.nvme0n1.in‚ 2244.94140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 31.59765625 disk.sda3.in‚0% disk./.usageGB‚1151.5417175292969% disk.nvme0n1.out‚19921.3076171875' disk.nvme0c0n1.out‚19921.3076171875 disk.dm-0.in‚ 2244.94140625# proc.memory.rssMB‚ 3262.15234375* proc.memory.percent‚19.910597801208496b"5_‚(:’ ÿËçÔö´ày gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚46.0 gpu.0.powerWatts‚253.331' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.666199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼oqÂ(:æ ÿËçÔÛ¶Ôz# disk.nvme0c0n1.in‚ 2244.94140625' disk.nvme0c0n1.out‚20318.6103515625 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.13671875 disk.sda2.out‚0 network.recv‚366325" disk.dm-0.out‚20318.6103515625! disk.nvme0n1.in‚ 2244.94140625 disk.sda2.in‚0$ memory_percent‚72.61767387390137( proc.memory.availableMB‚ 4486.3203125 disk.dm-1.out‚ 32.06640625 disk.sda3.in‚0 disk.sda3.out‚ 32.06640625% disk.nvme0n1.out‚20318.6103515625 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3262.15234375% disk./.usageGB‚1150.4520721435547 disk.dm-0.in‚ 2244.94140625 disk.dm-1.in‚0 disk.sda.out‚ 32.06640625* proc.memory.percent‚19.910597801208496 cpu‚23.923247537362343 proc.cpu.threads‚17 network.sent‚981694) disk./.usagePercent‚77.20780385513758#©wçÄ‚(:’ ŽÌçÔçì‘x gpu.0.gpu‚87.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚149.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.864800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ^GÈpÅ‚(:å ŽÌçÔБ‚{ disk.dm-0.in‚ 2244.94140625# disk.dm-0.out‚20511.33447265625 disk.dm-1.in‚0% proc.memory.availableMB‚ 4486.8125 cpu‚23.93880758927924) disk./.usagePercent‚77.20780385513758% disk./.usageGB‚1150.4520721435547 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 2244.94140625$ memory_percent‚72.61466979980469 network.recv‚370017 disk.sda3.out‚ 32.53515625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 2244.94140625( disk.nvme0c0n1.out‚20511.33447265625 disk.dm-1.out‚ 32.53515625# proc.memory.rssMB‚ 3262.15234375* proc.memory.percent‚19.910597801208496 disk.drbd1260.out‚ 0.13671875 disk.sda2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚988537 disk.sda.out‚ 32.53515625& disk.nvme0n1.out‚20511.33447265625/ 2Æ- _wandbruntime‚1019  _runtime‚1019}:|Ãǽ % lr-AdamW/pg1‚0.000349918951556065 % lr-AdamW/pg2‚0.000349918951556065 trainer/global_step‚649 % _timestamp‚1.7884708122809532e+09 _runtime‚1019.036742701 _step‚48d|ÙCÇÈÁ % lr-AdamW/pg1‚0.000349918951556065 % lr-AdamW/pg2‚0.000349918951556065 trainer/global_step‚649 % _timestamp‚1.7884708122809532e+09 _runtime‚1019.036742701 _step‚480àƒ›±2É- _wandbruntime‚1019  _runtime‚1019BÛL’xÊçO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16027286 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16002444 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13286295 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16027286 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29288739 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2167376984064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101567932862976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1568401094144 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SGDeviceStatsMonitor.on_train_batch_start/inactive_spli`fut.small_pool.freed‚5772514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100003168054272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5773218 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101571569148416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14097987 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14097278 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100003141662720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1564791200256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92755537061888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29313587 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90588160077824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1565672623660 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16002444 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100003168054272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚649 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13286301 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101567932862976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8324764 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99325771843964 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2167445494784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 % _timestamp‚1.7884708122819574e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 _runtime‚ 1019.03793042 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13286295 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚49KDÌ£ñ'ËëO RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚97763733303408 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100003141662720 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1564791200256 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1568401094144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99329405927068 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101567932862976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1568401094144 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5772514 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100003168054272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5773218 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101571569148416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14097987 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14097278 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100003141662720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1564791200256 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92755537061888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29313587 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90588160077824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1565672623660 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16002444 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100003168054272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚649 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13286301 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101567932862976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8324764 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99325771843964 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2167445494784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 % _timestamp‚1.7884708122819574e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 _runtime‚ 1019.03793042 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13286295 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚49 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16027286 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16002444 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13286295 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16027286 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29288739 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2167376984064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16051879 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5782134 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92898168625664 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101727757202432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1567201681920 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29358604 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13306719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2170718158848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 _step‚50 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101724120903680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1570811589120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16027011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16051879 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29358604 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29333730 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97914062544544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2170788753408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99478507388386 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29333730 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14119017 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92898290423296 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16027011 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1567201681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 % _timestamp‚1.7884708133733492e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13306719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 trainer/global_step‚649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100156919221760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90727450466816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568078927154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16051879 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5782134 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92898168625664 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101727757202432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1567201681920 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29358604 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13306719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2170718158848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 _step‚50 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101724120903680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1570811589120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16027011 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16051879 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29358604 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29333730 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97914062544544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2170788753408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99478507388386 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29333730 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14119017 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92898290423296 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16027011 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1567201681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 % _timestamp‚1.7884708133733492e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13306719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 trainer/global_step‚649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100156919221760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90727450466816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568078927154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚37172ó¶Y2Ñ- _wandbruntime‚1020  _runtime‚1020ÌuŸWÒÑ 1loss/SRqqt4/detection_loss‚3.5664722522099814 /loss/FBt2/detection_loss‚1.2497854008659965 /loss/FBt4/detection_loss‚2.8094465701203597 %loss/total_loss‚74.29613045578884 _runtime‚1020.131034009 /loss/FRt1/detection_loss‚1.9185289434798352 /loss/FRt1/assignment_loss‚5.331002343784679 0loss/FBt3/assignment_loss‚0.6203539419678307 epoch‚0 /loss/FRt2/detection_loss‚2.3260850529921684 /loss/FRt3/detection_loss‚3.0999941740717207 trainer/global_step‚649 /loss/FRt2/assignment_loss‚6.852919904809249 % _timestamp‚1.7884708133754423e+09 1loss/SRqqt3/assignment_loss‚3.266653115550677 0loss/FBt2/assignment_loss‚0.4242724301951907 1loss/SRqqt3/detection_loss‚3.5934985416630902 _step‚51 /loss/FRt3/assignment_loss‚6.380369458879743 2loss/SRqqt1/assignment_loss‚2.8569603157043457 2loss/SRqqt4/assignment_loss‚3.5981282869974773 /loss/FRt4/detection_loss‚2.7132344410774554 /loss/FBt1/detection_loss‚3.2802229492287887 0loss/FBt1/assignment_loss‚0.6284238488015513 .loss/FBt3/detection_loss‚4.470912245603708 0loss/FBt4/assignment_loss‚0.3698733589011873 /loss/FRt4/assignment_loss‚7.187274699515485 2loss/SRqqt2/assignment_loss‚3.2207648621665106 1loss/SRqqt1/detection_loss‚1.0286664889752866 1loss/SRqqt2/detection_loss‚3.5022868282265134œ1¸3[ÓÕ 1loss/SRqqt1/detection_loss‚1.0286664889752866 1loss/SRqqt2/detection_loss‚3.5022868282265134 1loss/SRqqt4/detection_loss‚3.5664722522099814 /loss/FBt2/detection_loss‚1.2497854008659965 /loss/FBt4/detection_loss‚2.8094465701203597 %loss/total_loss‚74.29613045578884 _runtime‚1020.131034009 /loss/FRt1/detection_loss‚1.9185289434798352 /loss/FRt1/assignment_loss‚5.331002343784679 0loss/FBt3/assignment_loss‚0.6203539419678307 epoch‚0 /loss/FRt2/detection_loss‚2.3260850529921684 /loss/FRt3/detection_loss‚3.0999941740717207 trainer/global_step‚649 /loss/FRt2/assignment_loss‚6.852919904809249 % _timestamp‚1.7884708133754423e+09 1loss/SRqqt3/assignment_loss‚3.266653115550677 0loss/FBt2/assignment_loss‚0.4242724301951907 1loss/SRqqt3/detection_loss‚3.5934985416630902 _step‚51 /loss/FRt3/assignment_loss‚6.380369458879743 2loss/SRqqt1/assignment_loss‚2.8569603157043457 2loss/SRqqt4/assignment_loss‚3.5981282869974773 /loss/FRt4/detection_loss‚2.7132344410774554 /loss/FBt1/detection_loss‚3.2802229492287887 0loss/FBt1/assignment_loss‚0.6284238488015513 .loss/FBt3/detection_loss‚4.470912245603708 0loss/FBt4/assignment_loss‚0.3698733589011873 /loss/FRt4/assignment_loss‚7.187274699515485 2loss/SRqqt2/assignment_loss‚3.22076486216651063 „¢ª›Ô‚(: ¬ÌçÔ÷¬Øx gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07À’$lÕ‚(:á ¬ÌçÔˆò’} disk.sda1.out‚0$ memory_percent‚72.61672019958496$ disk./.usageGB‚1150.475715637207 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 4486.4765625# proc.memory.rssMB‚ 3262.15234375( disk./.usagePercent‚77.2093905898348! disk.nvme0c0n1.in‚ 2380.109375 disk.sda3.out‚ 33.58984375 disk.drbd1260.out‚0.140625* proc.memory.percent‚19.910597801208496 network.recv‚379040 disk.sda.in‚0 disk.sda.out‚ 33.58984375 disk.nvme0n1.in‚ 2380.109375% disk.nvme0n1.out‚21087.2880859375 cpu‚23.968632464783155 proc.cpu.threads‚17 network.sent‚ 1.04301e+06' disk.nvme0c0n1.out‚21087.2880859375 disk.dm-0.in‚ 2380.109375" disk.dm-0.out‚21087.2880859375 disk.dm-1.out‚ 33.58984375 disk.sda2.in‚0ˆ¸6hÖ‚(:Ý »ÌçÔñ¢Ÿy( disk.nvme0c0n1.out‚21184.10595703125 disk.dm-1.out‚ 34.0546875 disk.sda.in‚0 disk.sda.out‚ 34.0546875 disk.sda2.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 3262.15625* proc.memory.percent‚19.910621643066406 network.sent‚ 1.0699e+06) disk./.usagePercent‚77.20261125432059$ disk./.usageGB‚1150.374698638916 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 2380.109375 cpu‚23.982952148901614 proc.cpu.threads‚17 network.recv‚384985! disk.nvme0c0n1.in‚ 2380.109375 disk.dm-0.in‚ 2380.109375 disk.sda3.in‚0& disk.nvme0n1.out‚21184.10595703125 disk.sda2.in‚0 disk.sda1.out‚0# disk.dm-0.out‚21184.10595703125 disk.dm-1.in‚0 disk.sda3.out‚ 34.0546875 disk.drbd1260.out‚ 0.1484375$ memory_percent‚72.61676788330078& proc.memory.availableMB‚ 4486.46875ä€kԒׂ(:‡ »ÌçÔ‹óÍy gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚47.0 gpu.0.powerWatts‚246.596' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.3192 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03óK…rØ‚(:ç ÊÌçÔ„Íšy( proc.memory.availableMB‚ 4486.7109375# proc.memory.rssMB‚ 3262.16015625 disk.nvme0n1.in‚ 2380.109375& disk.nvme0n1.out‚21590.09423828125 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 2380.109375* proc.memory.percent‚19.910645484924316 cpu‚23.996953731842353$ disk./.usageGB‚1150.465534210205 disk.dm-1.out‚ 34.74609375 disk.sda.in‚0 disk.sda.out‚ 34.74609375 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚ 1.076669e+06 network.recv‚388219) disk./.usagePercent‚77.20870730572592 disk.dm-1.in‚0( disk.nvme0c0n1.out‚21590.09423828125# disk.dm-0.out‚21590.09423828125$ memory_percent‚72.61528968811035 disk.sda3.in‚0 disk.sda3.out‚ 34.74609375 disk.drbd1260.out‚ 0.1484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 2380.109375ørè’Ù‚(:‡ ÊÌçÔ„ÌÆz gpu.0.gpu‚65.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚207.538' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C¿Ã[œÚ‚(:‘ ÙÌçÔÛÒçx gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚191.187' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚38.23740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0θs,cÛ‚(:Ø ÙÌçÔ±©Äz disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚72.61683940887451) proc.memory.availableMB‚ 4486.45703125# proc.memory.rssMB‚ 3262.16015625 cpu‚24.010754148295714) disk./.usagePercent‚77.20870960979373% disk./.usageGB‚1150.4655685424805$ disk.nvme0n1.out‚21951.337890625 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 2380.109375 disk.dm-0.in‚ 2380.109375! disk.dm-0.out‚21951.337890625 disk.dm-1.out‚34.9375 network.recv‚392912& disk.nvme0c0n1.out‚21951.337890625 disk.sda.out‚34.9375 disk.drbd1260.out‚ 0.16796875* proc.memory.percent‚19.910645484924316 proc.cpu.threads‚17 network.sent‚ 1.080876e+06 disk.nvme0n1.in‚ 2380.109375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚34.9375_ d¬2Ü- _wandbruntime‚1093  _runtime‚1093øz„Äݾ & lr-AdamW/pg1‚0.0003768772683169329 & lr-AdamW/pg2‚0.0003768772683169329 trainer/global_step‚699 $ _timestamp‚1.788470886422325e+09 _runtime‚1093.178033999 _step‚52‹ˆilÈÞ & lr-AdamW/pg1‚0.0003768772683169329 & lr-AdamW/pg2‚0.0003768772683169329 trainer/global_step‚699 $ _timestamp‚1.788470886422325e+09 _runtime‚1093.178033999 _step‚524+{ýæ2ß- _wandbruntime‚1093  _runtime‚1093ûô@ò'àìO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6217393 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106962562656272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1686001379168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1685329697280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31566489 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109377349312000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107692019614720 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107692046006272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17258988 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 % _timestamp‚1.7884708864231136e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31541641 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109377349312000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105280169932240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17258988 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99886371110400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107692019614720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14307495 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107692046006272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15181712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109380985597440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14307501 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2333764288000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 _step‚53 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17234146 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024ôK;ö'áðO RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 % _timestamp‚1.7884708864231136e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31541641 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109377349312000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105280169932240 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123908096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17258988 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99886371110400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107692019614720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14307495 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107692046006272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚700 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15181712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109380985597440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14307501 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2333764288000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 _step‚53 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17234146 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6217393 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106962562656272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1686001379168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚72705024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1685329697280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31566489 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109377349312000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107692019614720 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107692046006272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17258988 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚05_Þ •2â- _wandbruntime‚1094  _runtime‚1094sÙäÉãøM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107845823565312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14327919 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14327919 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 T@DeviceStatsM¶ÖÝà5$onitor.on_train_batch_end/active_bytes.all.allocated‚109537173651456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17283581 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8977819 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31586632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14327925 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107118932284006 trainer/global_step‚699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14327925 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15203241 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97691969916416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1684799027318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17258713 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100028997904384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107115298200694 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17258713 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1687740178944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109533537352704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884708874539042e+09 _step‚54 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109537173651456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31611506 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17283581 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1691350086144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15203917 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 _runtime‚1094.210333348 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6226093 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97692021119488 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31611506 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6225422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100029119702016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968%V·'äüM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31611506 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6225422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100029119702016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107845823565312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14327919 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14327919 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109537173651456 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17283581 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8977819 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31586632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14327925 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107118932284006 trainer/global_step‚699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14327925 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15203241 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97691969916416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1684799027318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17258713 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚100028997904384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107115298200694 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17258713 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1687740178944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109533537352704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7884708874539042e+09 _step‚54 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109537173651456 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31611506 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17283581 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1691350086144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15203917 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 _runtime‚1094.210333348 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6226093 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97692021119488 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117186765761024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114602987551684 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115380897566720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15328695 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9605264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _step‚57 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107017340233728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18465848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117190402046464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15328701 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1805868194304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16264547 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117190402046464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115380897566720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9605269 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115380923958272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500209767424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1806330134676 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15328701 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18490690 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18490690 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16265241 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33794543 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104517198977024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112796631989040ÙCú'ùôO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115380923958272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500209767424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1806330134676 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15328701 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18490690 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18490690 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16265241 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33794543 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104517198977024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112796631989040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚114599353468580 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1809478088192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1809478088192 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117186765761024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114602987551684 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115380897566720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15328695 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9605264 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _step‚57 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107017340233728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18465848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117190402046464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15328701 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1805868194304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16264547 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117190402046464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115380897566720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9605269 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚09TüÓu2ú- _wandbruntime‚1180  _runtime‚1180@½Y‡ÿ&ûùM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18515283 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117346590100480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15349119 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6667988 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107160097501696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117342953801728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚114755723096314 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1811888583168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33864408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6668658 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 $ _timestamp‚1.788470973613088e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15349125 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2503556932608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114752089013002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9618074 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112946961230176 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33864408 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1808736438170 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1805127782826 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1811888583168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107159975704064 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16286057 _runtime‚1180.369417218 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115534675125760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33839534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104656540569088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104656489366016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9618069 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 trainer/global_step‚749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15349119 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 _step‚58 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33839534 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2503486338048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200“i9× üýM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1808278675968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117346590100480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚107159975704064 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16286057 _runtime‚1180.369417218 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115534675125760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33839534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104656540569088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104656489366016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9618069 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 trainer/global_step‚749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15349119 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚670 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 _step‚58 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33839534 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2503486338048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18515283 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117346590100480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15349119 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6667988 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107160097501696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117342953801728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚114755723096314 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1811888583168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33864408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6668658 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 $ _timestamp‚1.788470973613088e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15349125 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2503556932608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114752089013002 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9618074 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112946961230176 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33864408 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1808736438170 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1805127782826 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1811888583168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124996182210048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19697550 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10245514 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚111481718426624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120313119473808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 _step‚61 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123069801910272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123069775518720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36072293 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19722392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2666871518208 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111481769629696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114148713852928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1930016585216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666944223232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124999818495488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17350219 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10245519 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122236144280888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124999818495488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 trainer/global_step‚799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7103973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1926658890184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16349895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36047445 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7104700 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122239778363992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1923050235048 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123069801910272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1930016585216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120313094045840 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111481769629696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114148713852928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1930016585216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666944223232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124999818495488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17350219 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10245519 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122236144280888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124999818495488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 trainer/global_step‚799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7103973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1926658890184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16349895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36047445 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7104700 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122239778363992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1923050235048 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123069801910272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1930016585216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1203130Å¡÷ê^94045840 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124996182210048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19697550 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10245514 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚111481718426624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120313119473808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 _step‚61 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123069801910272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123069775518720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36072293 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19722392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2666871518208 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17371202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10258319 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19722117 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125156006549504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125152370250752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16370319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10258324 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114291201360384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928817172992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111621008815616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884710615351634e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123223579469312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17371930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111621060018688 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 trainer/global_step‚799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36092436 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19746985 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1932427080192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16370319 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36117310 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚728 _step‚62 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114291325255168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36092436 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1929065193678 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36117310 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123223553077760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7112883 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16370325 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928817172992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚723 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122392513908622 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1925456538334 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36117310 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123223553077760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7112883 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16370325 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928817172992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚723 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122392513908622 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1925456538334 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17371202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10258319 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19722117 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125156006549504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125152370250752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16370319 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10258324 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114291201360384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928817172992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111621008815616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884710615351634e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123223579469312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17371930 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111621060018688 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 trainer/global_step‚799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36092436 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19746985 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1932427080192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16370319 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36117310 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚728 _step‚62 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114291325255168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123894784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36092436 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1929065193678 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532>Fÿó2•- _wandbruntime‚1268  _runtime‚1268νÌT–Î % _timestamp‚1.7884710615419505e+09 /loss/FRt2/assignment_loss‚6.326960337789435 1loss/FBt2/assignment_loss‚0.33657044562997057 /loss/FRt1/detection_loss‚1.8929477974142828 1loss/SRqqt4/detection_loss‚2.8653065231111317 trainer/global_step‚799 /loss/FRt4/assignment_loss‚6.066033372692034 2loss/SRqqt4/assignment_loss‚2.8192908278218023 0loss/FBt1/assignment_loss‚0.5677148474347968 1loss/SRqqt3/detection_loss‚2.8693923626257027 /loss/FBt4/detection_loss‚3.6466704607009888 _step‚63 epoch‚0 -loss/FRt2/detection_loss‚2.35405475214908 .loss/FRt3/detection_loss‚3.160514146089554 0loss/SRqqt1/detection_loss‚0.857884956653728 .loss/FBt3/detection_loss‚4.671411633491516 %loss/total_loss‚68.24064602934263 0loss/FRt1/assignment_loss‚4.7208622576485215 0loss/FBt4/assignment_loss‚0.6750839292967262 /loss/FRt4/detection_loss‚2.3638054620401534 _runtime‚1268.298806889 2loss/SRqqt2/assignment_loss‚2.8212286784098697 0loss/FBt3/assignment_loss‚0.5578936129808426 0loss/SRqqt2/detection_loss‚3.407444940163539 /loss/FBt2/detection_loss‚0.8344887260269994 /loss/FRt3/assignment_loss‚6.341003213609968 2loss/SRqqt1/assignment_loss‚2.6336951015531556 2loss/SRqqt3/assignment_loss‚2.9201568727907925 .loss/FBt1/detection_loss‚2.530230771218027€kÐX—Ò epoch‚0 -loss/FRt2/detection_loss‚2.35405475214908 .loss/FRt3/detection_loss‚3.160514146089554 0loss/SRqqt1/detection_loss‚0.857884956653728 .loss/FBt3/detection_loss‚4.671411633491516 %loss/total_loss‚68.24064602934263 0loss/FRt1/assignment_loss‚4.7208622576485215 0loss/FBt4/assignment_loss‚0.6750839292967262 /loss/FRt4/detection_loss‚2.3638054620401534 _runtime‚1268.298806889 2loss/SRqqt2/assignment_loss‚2.8212286784098697 0loss/FBt3/assignment_loss‚0.5578936129808426 0loss/SRqqt2/detection_loss‚3.407444940163539 /loss/FBt2/detection_loss‚0.8344887260269994 /loss/FRt3/assignment_loss‚6.341003213609968 2loss/SRqqt1/assignment_loss‚2.6336951015531556 2loss/SRqqt3/assignment_loss‚2.9201568727907925 .loss/FBt1/detection_loss‚2.530230771218027 % _timestamp‚1.7884710615419505e+09 /loss/FRt2/assignment_loss‚6.326960337789435 1loss/FBt2/assignment_loss‚0.33657044562997057 /loss/FRt1/detection_loss‚1.8929477974142828 1loss/SRqqt4/detection_loss‚2.8653065231111317 trainer/global_step‚799 /loss/FRt4/assignment_loss‚6.066033372692034 2loss/SRqqt4/assignment_loss‚2.8192908278218023 0loss/FBt1/assignment_loss‚0.5677148474347968 1loss/SRqqt3/detection_loss‚2.8693923626257027 /loss/FBt4/detection_loss‚3.6466704607009888 _step‚63?…J¿œ˜‚(:‘ œÎçÔͨßw gpu.0.gpu‚92.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚238.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.772000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h`gºf™‚(:Û œÎçÔ””ùx disk.sda1.out‚0 disk.sda.out‚41.71875' proc.memory.availableMB‚ 4485.578125 proc.cpu.threads‚17) disk./.usagePercent‚77.30187228754897$ disk./.usageGB‚1151.853759765625 disk.sda2.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚25883.8603515625 disk.dm-1.out‚41.71875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1953125 network.sent‚ 1.240338e+06 network.recv‚446742% disk.nvme0n1.out‚25883.8603515625" disk.nvme0c0n1.in‚ 2781.7109375" disk.dm-0.out‚25883.8603515625 disk.dm-1.in‚0 disk.sda3.out‚41.71875# memory_percent‚72.6222038269043 disk.nvme0n1.in‚ 2781.7109375 disk.sda2.in‚0 disk.dm-0.in‚ 2781.7109375 disk.sda.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 cpu‚24.16445287657412ÕH`“œš‚(:‘ «ÎçÔƒ „y gpu.0.gpu‚16.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚39.0 gpu.0.powerWatts‚166.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.312799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö†¡i›‚(:Þ «ÎçÔ¡Ç y network.recv‚453064$ disk./.usageGB‚1151.978744506836 disk.dm-1.in‚0 disk.sda3.out‚ 42.12890625 disk.nvme0n1.in‚ 2937.921875( proc.memory.availableMB‚ 4486.0234375* proc.memory.percent‚19.910645484924316 network.sent‚ 1.292486e+06 disk.sda.out‚ 42.12890625 disk.sda3.in‚0# disk.nvme0n1.out‚26135.21484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚72.61948585510254) disk./.usagePercent‚77.31026011840082! disk.nvme0c0n1.in‚ 2937.921875 disk.dm-1.out‚ 42.12890625 disk.sda1.out‚0# proc.memory.rssMB‚ 3262.16015625 cpu‚24.17438604868063 proc.cpu.threads‚17% disk.nvme0c0n1.out‚26135.21484375 disk.dm-0.in‚ 2937.921875 disk.dm-0.out‚26135.21484375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.19921875º£¶^œ‚(:’ ºÎçÔÿöþx gpu.0.gpu‚77.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚144.053' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.810599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ë‚s‚(:è ºÎçÔ”Ášy( disk.nvme0c0n1.out‚26419.91064453125 disk.dm-0.in‚ 2937.921875# proc.memory.rssMB‚ 3262.16015625 cpu‚24.184297683821022( disk./.usagePercent‚77.3271074622141 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0# disk.dm-0.out‚26419.91064453125) proc.memory.availableMB‚ 4486.01953125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 42.58203125 disk.sda3.in‚0 disk.sda3.out‚ 42.58203125 disk.drbd1260.in‚0! disk.nvme0c0n1.in‚ 2937.921875$ memory_percent‚72.61950969696045 network.sent‚ 1.29545e+06% disk./.usageGB‚1152.2297821044922 disk.drbd1260.out‚ 0.19921875 disk.sda2.out‚0 disk.sda1.in‚0* proc.memory.percent‚19.910645484924316& disk.nvme0n1.out‚26419.91064453125 network.recv‚456381 disk.dm-1.out‚ 42.58203125 disk.nvme0n1.in‚ 2937.921875 &ý’ž‚(:‡ ÉÎçÔ“øw gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚45.0 gpu.0.powerWatts‚276.283' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸mÁqŸ‚(:æ ÉÎçÔª¦ày disk.sda3.out‚ 43.12890625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.203125 disk.sda2.in‚0 network.recv‚460279 disk.nvme0n1.in‚ 2937.921875& disk.nvme0n1.out‚26768.63134765625 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 2937.921875( disk.nvme0c0n1.out‚26768.63134765625 disk.dm-0.in‚ 2937.921875# disk.dm-0.out‚26768.63134765625 disk.sda2.out‚0 disk.sda.in‚0) proc.memory.availableMB‚ 4486.26171875# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 cpu‚24.193977769155858 proc.cpu.threads‚17 network.sent‚ 1.30149e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 43.12890625 disk.sda.out‚ 43.12890625$ memory_percent‚72.61803150177002 disk.sda1.in‚0) disk./.usagePercent‚77.32703782816479$ disk./.usageGB‚1152.228744506836 disk.sda3.in‚0Ï5¢}› ‚(: ØÎçÔ»ÕÂz gpu.0.gpu‚23.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚39.0 gpu.0.powerWatts‚167.09' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.418000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}¯ÍL^¡‚(:á ØÎçÔ« ±{# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.9106DÄÆ45484924316 cpu‚24.20324290667659 network.sent‚ 1.305682e+06) disk./.usagePercent‚77.32704064424767 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 2937.921875 disk.dm-1.in‚0 disk.dm-1.out‚ 43.91796875 disk.sda.out‚ 43.91796875% disk./.usageGB‚1152.2287864685059 disk.sda3.out‚ 43.91796875 disk.drbd1260.out‚0.203125$ disk.nvme0n1.out‚27103.044921875 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 2937.921875! disk.dm-0.out‚27103.044921875) proc.memory.availableMB‚ 4486.01953125 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 2937.921875 proc.cpu.threads‚17 network.recv‚463554 disk.sda1.in‚0& disk.nvme0c0n1.out‚27103.044921875 disk.sda.in‚0$ memory_percent‚72.61950969696045Z?ˆ^¢‚(:Ó çÎçÔ…­‚x disk.sda3.in‚0# proc.memory.rssMB‚ 3262.16015625 disk.drbd1260.out‚ 0.20703125& disk.nvme0n1.out‚27546.76025390625 disk.sda.out‚ 44.38671875$ memory_percent‚72.61826992034912) proc.memory.availableMB‚ 4486.22265625 cpu‚24.21266163464431 proc.cpu.threads‚17 network.recv‚468213$ disk./.usageGB‚1152.229206085205 disk.dm-0.in‚3067 disk.dm-1.out‚ 44.38671875* proc.memory.percent‚19.910645484924316 disk.drbd1260.in‚0 disk.nvme0n1.in‚3067 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚27546.76025390625 disk.dm-1.in‚0 disk.sda3.out‚ 44.38671875 network.sent‚ 1.312215e+06) disk./.usagePercent‚77.32706880507644 disk.sda2.out‚0 disk.nvme0c0n1.in‚3067# disk.dm-0.out‚27546.76025390625 disk.sda.in‚0Ñs§‘£‚(:† çÎçÔÃñ¬x gpu.0.gpu‚26.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.354' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò¯É2¤- _wandbruntime‚1351  _runtime‚1351:µ[>Ä¥¾ " _timestamp‚1.7884711451221e+09 _runtime‚1351.879313771 _step‚64 ' lr-AdamW/pg1‚0.00045775221859953655 ' lr-AdamW/pg2‚0.00045775221859953655 trainer/global_step‚849És8Ȧ ' lr-AdamW/pg1‚0.00045775221859953655 ' lr-AdamW/pg2‚0.00045775221859953655 trainer/global_step‚849 " _timestamp‚1.7884711451221e+09 _runtime‚1351.879313771 _step‚64@`¥6ƒ2§- _wandbruntime‚1351  _runtime‚1351>ø'¨òO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18435404 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130758653470720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132805598659072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2833428221952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127829556102640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚132809234944512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118446289079296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10885764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2046945188352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38325195 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20929252 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121279666098176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚132809234944512 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20929252 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18434695 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38300347 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚130758653470720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10885769 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121279785811968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2050555082240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129876569176300 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚130758679862272 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121279666098176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚132809234944512 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20929252 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18434695 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38300347 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚130758653470720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10885769 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121279785811968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2050555082240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129876569176300 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚130758679862272 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132805598659072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2833428221952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127829556102640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚132809234944512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118446289079296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0A’MÚË2ª- _wandbruntime‚1352  _runtime‚1352å_“Ù %«úM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2052965577216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚132965422998528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118585528265216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17391519 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7557831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118585579468288 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130912457421312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17391525 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10898574 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7558514 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚849 _runtime‚1352.911975306 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38370212 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20953819 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121422295141376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2836766876160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 _step‚66 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130912457421312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049355670016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130025670637618 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049355670016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10898569 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18456400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121422416939008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78847114615464e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17391519 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18457088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127979885343776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38370212 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20953819 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚132961786699776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17391525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127979910771744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130912457421312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17391525 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10898574 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7558514 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚849 _runtime‚1352.911975306 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38370212 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20953819 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121422295141376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2836766876160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 _step‚66 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130912457421312 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049355670016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130025670637618 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049355670016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10898569 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18456400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121422416939008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 # _timestamp‚1.78847114615464e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17391519 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18457088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127979885343776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38370212 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20953819 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚132961786699776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17391525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127979910771744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚132965422998528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118585528265216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17391519 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7557831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118585579468288 Bè{*[²‚(:Ð …ÏçÔ伫{ disk.sda3.out‚ 45.4140625 disk.sda2.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚3067$ memory_percent‚72.62167930603027 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3262.16015625 cpu‚24.229736340983898 disk.drbd1260.out‚ 0.2109375 network.recv‚477227) disk./.usagePercent‚77.25552621959531# disk.dm-0.out‚28206.33349609375 disk.dm-1.out‚ 45.4140625 disk.sda.in‚0 disk.nvme0n1.in‚3067& disk.nvme0n1.out‚28206.33349609375 network.sent‚ 1.366893e+06% disk./.usageGB‚1151.1631698608398 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚28206.33349609375( proc.memory.availableMB‚ 4485.6640625* proc.memory.percent‚19.910645484924316 disk.dm-0.in‚3067 disk.sda.out‚ 45.4140625v×™³‚(:’ …ÏçÔ­¯óy gpu.0.gpu‚85.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚45.0 gpu.0.powerWatts‚174.715' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.943000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:]'´‚(:’ ”ÏçÔºõôx gpu.0.gpu‚32.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚47.0 gpu.0.powerWatts‚277.921' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.584199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ëåÄMµ‚(: ”ÏçÔìðÎ{ cpu‚24.238480660671044 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚3067# disk.dm-0.out‚28470.95849609375 disk.drbd1260.in‚0$ memory_percent‚72.61712551116943* proc.memory.percent‚19.910645484924316 network.sent‚ 1.373588e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚3067 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚46.25 network.recv‚480679) disk./.usagePercent‚77.28908163912325% disk./.usageGB‚1151.6631698608398 disk.nvme0n1.in‚3067( disk.nvme0c0n1.out‚28470.95849609375 disk.sda3.in‚0 disk.sda3.out‚46.25 disk.drbd1260.out‚ 0.2109375& disk.nvme0n1.out‚28470.95849609375 disk.dm-1.out‚46.25) proc.memory.availableMB‚ 4486.41015625# proc.memory.rssMB‚ 3262.160156253˜û붂(:’ £ÏçÔ÷£¡w gpu.0.gpu‚40.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.773' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.354600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0££ÏŒr·‚(:ç £ÏçÔ®´üy disk.dm-1.out‚ 46.7265625 disk.sda.out‚ 46.7265625 disk.sda3.in‚0) disk./.usagePercent‚77.28908445520614 disk.nvme0n1.in‚ 3241.4140625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.out‚ 46.7265625 disk.drbd1260.in‚0* proc.memory.percent‚19.910645484924316 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚28734.79931640625 disk.dm-0.in‚ 3241.4140625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.21484375$ memory_percent‚72.62020111083984# proc.memory.rssMB‚ 3262.16015625 network.recv‚485539# disk.dm-0.out‚28734.79931640625& proc.memory.availableMB‚ 4485.90625 cpu‚24.247283151272406 network.sent‚ 1.378108e+06% disk./.usageGB‚1151.6632118225098& disk.nvme0n1.out‚28734.79931640625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 3241.4140625çí2¸- _wandbruntime‚1414  _runtime‚1414à_¾5ƹÀ ' lr-AdamW/pg2‚0.00048471053536040446 trainer/global_step‚899 % _timestamp‚1.7884712077107954e+09 _runtime‚ 1414.46633796 _step‚68 ' lr-AdamW/pg1‚0.00048471053536040446ô¢“漀 ' lr-AdamW/pg2‚0.00048471053536040446 trainer/global_step‚899 % _timestamp‚1.7884712077107954e+09 _runtime‚ 1414.46633796 _step‚68 ' lr-AdamW/pg1‚0.00048471053536040446Dfo~§2»- _wandbruntime‚1414  _runtime‚1414Øš7ø'¼òO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138447557814272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19520377 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140615015108096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18392301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7994363 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135346018159440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2163707746064 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137509725905504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2171093579264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135346043587408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138447531422720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2171093579264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125410808528896 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7995062 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3000215919616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22160954 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚69 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22185796 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18392295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19521081 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22185796 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140618651393536 _runtime‚1414.467550786 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138447531422720 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11526019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128411095056384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121810944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3000286527488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40553249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11526014 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137513359988608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18392295 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128410973245440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2167483685376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138447557814272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125410757325824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2167316401200 trainer/global_step‚899 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18392301 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140615015108096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18392301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7994363 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135346018159440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2163707746064 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137509725905504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2171093579264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135346043587408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138447531422720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2171093579264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125410808528896 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7995062 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3000215919616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_den²^ú]vice_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22160954 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚69 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22185796 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18392295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19521081 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22185796 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140618651393536 _runtime‚1414.467550786 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138447531422720 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11526019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128411095056384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121810944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3000286527488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40553249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11526014 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137513359988608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18392295 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128410973245440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2167483685376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138447557814272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125410757325824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2167316401200 trainer/global_step‚899 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18392301 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138601335373312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19542627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135496372828544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128553601964032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18412719 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140774839447552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140771203148800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18412719 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125550047714816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125550098917888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138601308981760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2169894167040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40623114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11538824 _step‚70 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135496347400576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2166114049350 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2169894167040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19541944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8003125 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140774839447552 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3003622746624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138601335373312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68497408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8003803 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3003554249216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18412725 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40598240 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138601335373312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19542627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135496372828544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128553601964032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18412719 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140774839447552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140771203148800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18412719 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125550047714816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125550098917888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138601308981760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2169894167040 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40623114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11538824 _step‚70 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135496347400576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2166114049350 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2169894167040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19541944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8003125 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140774839447552 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3003622746624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138601335373312 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68497408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8003803 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3003554249216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18412725 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40598240 FHå¬2Á- _wandbruntime‚1415  _runtime‚1415ù¨'øOÂÉ .loss/FRt3/detection_loss‚3.282184820175171 0loss/SRqqt4/detection_loss‚2.455776813129584 2loss/SRqqt1/assignment_loss‚2.8385992436100254 /loss/FRt1/assignment_loss‚4.600223660469055 2loss/SRqqt2/assignment_loss‚3.9175693511962892 1loss/SRqqt3/assignment_loss‚3.078208009401957 .loss/FBt4/detection_loss‚5.335023194551468 $loss/total_loss‚74.6095511581334 trainer/global_step‚899 0loss/FRt4/assignment_loss‚5.8640532195568085 1loss/FBt2/assignment_loss‚0.23602246503417004 .loss/FRt2/detection_loss‚2.471638802377904 $ _timestamp‚1.788471208836035e+09 /loss/FRt4/detection_loss‚3.1033267662860453 0loss/SRqqt1/detection_loss‚0.803326586650001 0loss/SRqqt2/detection_loss‚5.443245458602905 /loss/FRt3/assignment_loss‚5.950765209197998 1loss/SRqqt4/assignment_loss‚2.928421940654516 /loss/FBt2/detection_loss‚0.7869838997844214 /loss/FBt3/detection_loss‚6.4246861934661865 epoch‚0 _step‚71 .loss/FRt2/assignment_loss‚5.06755764660288 0loss/FBt1/assignment_loss‚0.8350979499518871 .loss/FBt1/detection_loss‚2.525600105524063 0loss/FBt3/assignment_loss‚1.1057343482971191 0loss/FBt4/assignment_loss‚0.7448951544356532 /loss/FRt1/detection_loss‚1.8429984971880913 1loss/SRqqt3/detection_loss‚2.9676118219892182 _runtime‚1415.591464654H¹‰lSÃÍ 2loss/SRqqt1/assignment_loss‚2.8385992436100254 /loss/FRt1/assignment_loss‚4.600223660469055 2loss/SRqqt2/assignment_loss‚3.9175693511962892 1loss/SRqqt3/assignment_loss‚3.078208009401957 .loss/FBt4/detection_loss‚5.335023194551468 $loss/total_loss‚74.6095511581334 trainer/global_step‚899 0loss/FRt4/assignment_loss‚5.8640532195568085 1loss/FBt2/assignment_loss‚0.23602246503417004 .loss/FRt2/detection_loss‚2.471638802377904 $ _timestamp‚1.788471208836035e+09 /loss/FRt4/detection_loss‚3.1033267662860453 0loss/SRqqt1/detection_loss‚0.803326586650001 0loss/SRqqt2/detection_loss‚5.443245458602905 /loss/FRt3/assignment_loss‚5.950765209197998 1loss/SRqqt4/assignment_loss‚2.928421940654516 /loss/FBt2/detection_loss‚0.7869838997844214 /loss/FBt3/detection_loss‚6.4246861934661865 epoch‚0 _step‚71 .loss/FRt2/assignment_loss‚5.06755764660288 0loss/FBt1/assignment_loss‚0.8350979499518871 .loss/FBt1/detection_loss‚2.525600105524063 0loss/FBt3/assignment_loss‚1.1057343482971191 0loss/FBt4/assignment_loss‚0.7448951544356532 /loss/FRt1/detection_loss‚1.8429984971880913 1loss/SRqqt3/detection_loss‚2.9676118219892182 _runtime‚1415.591464654 .loss/FRt3/detection_loss‚3.282184820175171 0loss/SRqqt4/detection_loss‚2.455776813129584GTŸîú’Ä‚(:‡ ²ÏçÔµ÷¾| gpu.0.gpu‚60.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚188.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.6952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m–BxÅ‚(:í ²ÏçÔ¸Û÷ network.recv‚488568 disk.nvme0n1.in‚ 3241.4140625 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 3241.4140625 disk.dm-0.in‚ 3241.4140625 disk.dm-1.in‚0 disk.dm-1.out‚ 47.21484375$ memory_percent‚72.61903285980225 network.sent‚ 1.396048e+06) disk./.usagePercent‚77.28908215113833 disk.sda.out‚ 47.21484375 disk.sda3.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚29215.85400390625) proc.memory.availableMB‚ 4486.09765625 cpu‚24.255987172022444 disk.sda3.out‚ 47.21484375 disk.drbd1260.out‚ 0.21484375& disk.nvme0n1.out‚29215.85400390625 disk.sda2.in‚0 disk.sda1.out‚0% disk./.usageGB‚1151.6631774902344 disk.sda.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚29215.85400390625# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 proc.cpu.threads‚17»0{<‘Æ‚(:† ÁÏçÔ­ÿ’z gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.635' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.727 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìžÂ¶mÇ‚(:â ÁÏçÔÜÁŠ{ disk.sda1.out‚0( proc.memory.availableMB‚ 4486.0859375* proc.memory.percent‚19.910645484924316 cpu‚24.26601646191529 network.sent‚ 1.433168e+06 network.recv‚494406' disk.nvme0c0n1.out‚29361.7392578125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 47.60546875 disk.nvme0n1.in‚ 3241.4140625% disk.nvme0n1.out‚29361.7392578125& disk./.usagePercent‚77.28229078327% disk./.usageGB‚1151.5619812011719" disk.dm-0.out‚29361.7392578125 disk.dm-1.out‚ 47.60546875 disk.sda.out‚ 47.60546875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 3241.4140625 disk.drbd1260.out‚0.21875$ memory_percent‚72.61910438537598# proc.memory.rssMB‚ 3262.16015625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 3241.4140625 disk.sda2.in‚0 disk.sda2.out‚0)D9È‚(:Ú ÐÏçÔÕÊËx% disk.nvme0n1.out‚29733.6259765625 disk.sda2.in‚0$ memory_percent‚72.61905670166016& proc.memory.availableMB‚ 4486.09375* proc.memory.percent‚19.910645484924316 network.sent‚ 1.439836e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚29733.6259765625" disk.dm-0.out‚29733.6259765625 disk.sda3.out‚48.21875 disk.sda1.in‚0# proc.memory.rssMB‚ 3262.16015625 cpu‚24.27384088877067 proc.cpu.threads‚17 network.recv‚497467" disk.nvme0c0n1.in‚ 3241.4140625 disk.dm-0.in‚ 3241.4140625 disk.dm-1.out‚48.21875 disk.sda.in‚0 disk.sda.out‚48.21875 disk.drbd1260.out‚0.21875 disk.nvme0n1.in‚ 3241.4140625) disk./.usagePercent‚77.28774655983219% disk./.usagÁ 2/,eGB‚1151.6432762145996 disk.sda3.in‚0+Ù£æÉ‚(:… ÐÏçÔ µàx gpu.0.gpu‚53.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚46.0 gpu.0.powerWatts‚274.175' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.835 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A3gkÊ‚(:à ßÏçÔ³ •y network.recv‚502110" disk.dm-0.out‚30201.8798828125 disk.drbd1260.out‚ 0.22265625 disk.nvme0n1.in‚ 3290.1328125% disk.nvme0n1.out‚30201.8798828125 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚77.29319747225124% disk./.usageGB‚1151.7244987487793 disk.dm-0.in‚ 3290.1328125 disk.dm-1.in‚0 disk.dm-1.out‚48.78125 disk.sda3.out‚48.78125# proc.memory.rssMB‚ 3262.16015625 network.sent‚ 1.444248e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 3290.1328125) proc.memory.availableMB‚ 4485.80859375* proc.memory.percent‚19.910645484924316 disk.sda.out‚48.78125 disk.sda2.out‚0' disk.nvme0c0n1.out‚30201.8798828125# memory_percent‚72.6207971572876 cpu‚24.282532640353633 proc.cpu.threads‚17‚^ M‘Ë‚(:† ßÏçÔáÇy gpu.0.gpu‚32.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚187.522' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.5044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬¹ÈÌ‚(:… îÏçÔÿÄûx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gåqÍ‚(:æ îÏçÔ‡ö™z proc.cpu.threads‚17 disk.drbd1260.out‚ 0.22265625 network.sent‚ 1.449165e+06 disk.dm-1.out‚ 49.16015625 disk.sda3.in‚0 disk.sda3.out‚ 49.16015625% disk.nvme0n1.out‚30202.1689453125 cpu‚24.291836930671057) disk./.usagePercent‚77.29319875228892% disk./.usageGB‚1151.7245178222656" disk.dm-0.out‚30202.1689453125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 3365.484375 disk.sda2.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 3365.484375 disk.dm-0.in‚ 3365.484375 disk.sda.in‚0 disk.sda.out‚ 49.16015625 disk.drbd1260.in‚0 disk.sda1.in‚0$ memory_percent‚72.62082099914551( proc.memory.availableMB‚ 4485.8046875* proc.memory.percent‚19.910645484924316 network.recv‚505276 disk.sda2.in‚0' disk.nvme0c0n1.out‚30202.1689453125# proc.memory.rssMB‚ 3262.16015625¥gCú2Î- _wandbruntime‚1495  _runtime‚1495u-/ÂÃϽ _step‚72 & lr-AdamW/pg1‚0.0005116688521212723 & lr-AdamW/pg2‚0.0005116688521212723 trainer/global_step‚949 $ _timestamp‚1.788471288419055e+09 _runtime‚ 1495.17535329BûÚÇÐÁ _step‚72 & lr-AdamW/pg1‚0.0005116688521212723 & lr-AdamW/pg2‚0.0005116688521212723 trainer/global_step‚949 $ _timestamp‚1.788471288419055e+09 _runtime‚ 1495.17535329H=qÙ2Ñ- _wandbruntime‚1495  _runtime‚1495ô31‘ø'ÒòO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20606072 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42806151 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23417498 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2288022182400 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8439803 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3167056817664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146136409374720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8439123 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚135542265082368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148428067842560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23392656 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20605387 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3166988306944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19413501 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12166264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146136409374720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148424431557120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148424431557120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142862480216240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚73 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2291632076288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2287645156708 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2284036501572 % _timestamp‚1.7884712884224892e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148428067842560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 :ü'ÓöO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148424431557120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142862480216240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚73 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2291632076288 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2287645156708 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2284036501572 % _timestamp‚1.7884712884224892e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148428067842560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20606072 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42806151 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23417498 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2288022182400 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8439803 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3167056817664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146136409374720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8439123 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚135542265082368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148428067842560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23392656 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20605387 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3166988306944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19413501 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12166264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146136409374720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148424431557120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717IÑ’z02Ô- _wandbruntime‚1496  _runtime‚1496ÈP Ñ!ÕûM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146290213325312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23442091 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20627560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143012834885344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23417223 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148584255896576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12179069 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19433925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3170396078592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1496.485008964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132514618367488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2290051460202 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23417223 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135684892648448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433925 $ _timestamp‚1.788471289728281e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2294042571264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143012809457376 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20626892 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2290432664064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145299252262234 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42851142 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146290186933760 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12179074 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/ÇÞ#~0requested_bytes.all.peak‚ 28880497613 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135685014446080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42876016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8447823 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148584255896576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 trainer/global_step‚949 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145302886345546 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024C~à®'ÖÿM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146290186933760 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12179074 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135685014446080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42876016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8447823 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148584255896576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 trainer/global_step‚949 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145302886345546 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146290213325312 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23442091 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20627560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143012834885344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23417223 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148584255896576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12179069 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19433925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3170396078592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1496.485008964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132514618367488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2290051460202 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23417223 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135684892648448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433925 $ _timestamp‚1.788471289728281e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2294042571264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143012809457376 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20626892 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2290432664064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145299252262234 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42851142 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0J¥åãz2×- _wandbruntime‚1496  _runtime‚14967Ò(PØÊ /loss/FBt1/detection_loss‚2.9439602295557656 trainer/global_step‚949 0loss/FBt4/assignment_loss‚0.3856193128936828 epoch‚0 /loss/FBt2/detection_loss‚0.6751603119375528 _runtime‚1496.486777237 _step‚75 /loss/FRt1/assignment_loss‚3.361812324591086 /loss/FRt3/assignment_loss‚6.770625400543213 2loss/FBt3/assignment_loss‚0.000770391256082803 1loss/SRqqt1/detection_loss‚0.9086174479789204 0loss/SRqqt3/detection_loss‚3.224414575099945 -loss/FBt3/detection_loss‚4.82133948802948 /loss/FRt2/assignment_loss‚4.738018724051389 0loss/FBt2/assignment_loss‚0.3310810312829167 2loss/SRqqt3/assignment_loss‚2.7202168881893156 .loss/FRt1/detection_loss‚1.737378671781109 /loss/FRt3/detection_loss‚3.2295474147796632 /loss/FRt4/detection_loss‚2.8420869583884873 1loss/SRqqt4/detection_loss‚2.6876550992329915 .loss/FBt4/detection_loss‚3.063256931304932 %loss/total_loss‚67.07415477976751 $ _timestamp‚1.788471289731431e+09 /loss/FRt4/assignment_loss‚5.939827648798625 1loss/SRqqt1/assignment_loss‚2.242680112631233 2loss/SRqqt4/assignment_loss‚2.9129363695780435 .loss/FRt2/detection_loss‚2.584383948621425 /loss/SRqqt2/detection_loss‚4.25367538134257 1loss/SRqqt2/assignment_loss‚4.066500902175903 0loss/FBt1/assignment_loss‚0.63258921572317686" TÙÎ 1loss/SRqqt2/assignment_loss‚4.066500902175903 0loss/FBt1/assignment_loss‚0.6325892157231768 /loss/FBt1/detection_loss‚2.9439602295557656 trainer/global_step‚949 0loss/FBt4/assignment_loss‚0.3856193128936828 epoch‚0 /loss/FBt2/detection_loss‚0.6751603119375528 _runtime‚1496.486777237 _step‚75 /loss/FRt1/assignment_loss‚3.361812324591086 /loss/FRt3/assignment_loss‚6.770625400543213 2loss/FBt3/assignment_loss‚0.000770391256082803 1loss/SRqqt1/detection_loss‚0.9086174479789204 0loss/SRqqt3/detection_loss‚3.224414575099945 -loss/FBt3/detection_loss‚4.82133948802948 /loss/FRt2/assignment_loss‚4.738018724051389 0loss/FBt2/assignment_loss‚0.3310810312829167 2loss/SRqqt3/assignment_loss‚2.7202168881893156 .loss/FRt1/detection_loss‚1.737378671781109 /loss/FRt3/detection_loss‚3.2295474147796632 /loss/FRt4/detection_loss‚2.8420869583884873 1loss/SRqqt4/detection_loss‚2.6876550992329915 .loss/FBt4/detection_loss‚3.063256931304932 %loss/total_loss‚67.07415477976751 $ _timestamp‚1.788471289731431e+09 /loss/FRt4/assignment_loss‚5.939827648798625 1loss/SRqqt1/assignment_loss‚2.242680112631233 2loss/SRqqt4/assignment_loss‚2.9129363695780435 .loss/FRt2/detection_loss‚2.584383948621425 /loss/SRqqt2/detection_loss‚4.25367538134257KÐP!ánÚ‚(:ã ýÏçÔÿèùx network.sent‚ 1.455268e+06 network.recv‚509665 disk.sda3.out‚ 49.6640625 disk.drbd1260.out‚ 0.2265625& disk.nvme0n1.out‚30419.16748046875 disk.sda1.out‚0( proc.memory.availableMB‚ 4485.0390625) disk./.usagePercent‚77.29319747225124( disk.nvme0c0n1.out‚30419.16748046875 disk.dm-0.in‚ 3365.484375 disk.dm-1.out‚ 49.6640625 disk.sda.out‚ 49.6640625 disk.nvme0n1.in‚ 3365.484375 disk.sda1.in‚0 cpu‚24.30025459486105# disk.dm-0.out‚30419.16748046875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0# memory_percent‚72.6254940032959# proc.memory.rssMB‚ 3262.16015625% disk./.usageGB‚1151.7244987487793! disk.nvme0c0n1.in‚ 3365.484375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.out‚0* proc.memory.percent‚19.910645484924316 proc.cpu.threads‚17ŒG|‚‘Û‚(:† ýÏçÔº…•y gpu.0.gpu‚35.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚43.0 gpu.0.powerWatts‚251.746' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.3492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gâ+’Ü‚(:‡ ŒÐçÔ´Ýx gpu.0.gpu‚97.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚40.0 gpu.0.powerWatts‚127.607' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gfÍuiÝ‚(:Þ ŒÐçÔ¡¾èy disk.sda2.out‚0# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316% disk./.usageGB‚1151.7292938232422 disk.sda2.in‚0( disk.nvme0c0n1.out‚30735.85498046875# disk.dm-0.out‚30735.85498046875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚50.15625 disk.drbd1260.in‚0 network.recv‚514611 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 3365.484375 disk.dm-1.out‚50.15625 disk.sda.out‚50.15625 disk.sda3.in‚0$ memory_percent‚72.61967658996582) disk./.usagePercent‚77.29351927372178 disk.dm-0.in‚ 3365.484375 disk.drbd1260.out‚ 0.2265625( proc.memory.availableMB‚ 4485.9921875 cpu‚24.308759163306433 proc.cpu.threads‚17 network.sent‚ 1.50427e+06 disk.nvme0n1.in‚ 3365.484375& disk.nvme0n1.out‚30735.85498046875²"&kÞ‚(:à ›ÐçÔï÷x" disk.dm-0.out‚31072.7548828125 disk.dm-1.out‚ 50.546875 disk.sda.out‚ 50.546875 disk.sda3.in‚0 disk.sda3.out‚ 50.546875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 3365.484375% disk.nvme0n1.out‚31072.7548828125 disk.sda2.out‚0 disk.drbd1260.out‚ 0.23046875# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 cpu‚24.317623460065253 proc.cpu.threads‚17% disk./.usageGB‚1151.7919807434082' disk.nvme0c0n1.out‚31072.7548828125 disk.dm-0.in‚ 3365.484375 disk.sda.in‚0 network.sent‚ 1.510679e+06) disk./.usagePercent‚77.29772624553195 disk.sda2.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 3365.484375 disk.dm-1.in‚0$ memory_percent‚72.61967658996582( proc.memory.availableMB‚ 4485.9921875 network.recv‚519576 disk.sda1.in‚0D‰9ô’ß‚(:‡ ›ÐçÔÐýÑy gpu.0.gpu‚55.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚216.629' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.3258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sî>‘à‚(:† ªÐçÔ€ÞÂy gpu.0.gpu‚45.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚40.0 gpu.0.powerWatts‚174.066' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Ïüná‚(:ã ªÐçÔ˜ÈØz# proc.memory.rssMB‚ 3262.16015625 network.recv‚522909 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚72.61970043182373* proc.memory.percent‚19.910645484924316 cpu‚24.32639665213751 proc.cpu.threads‚17 disk.sda3.out‚ 50.9453125) disk./.usagePercent‚77.30190275244556! disk.nvme0c0n1.in‚ 3484.015625' disk.nvme0c0n1.out‚31382.4462890625 disk.dm-1.in‚0 disk.sda.out‚ 50.9453125 disk.sda3.in‚0 disk.nvme0n1.in‚ 3484.015625 network.sent‚ 1.515498e+06% disk./.usageGB‚1151.8542137145996" disk.dm-0.out‚31382.4462890625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.23046875 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 3484.015625 disk.dm-1.out‚ 50.9453125% disk.nvme0n1.out‚31382.4462890625) proc.memory.availableMB‚ 4485.98828125‰eTR’â‚(:‡ ¹ÐçÔ¡…Êy gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚240.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.0346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08#ýJkã‚(:à ¹ÐçÔÊȧz! disk.nvme0c0n1.in‚ 3484.015625' disk.nvme0c0n1.out‚31825.9248046875$ memory_percent‚72.62122631072998* proc.memory.percent‚19.910645484924316 cpu‚24.33431990771731 proc.cpu.threads‚17 network.sent‚ 1.521601e+06 disk.nvme0n1.in‚ 3484.015625% disk.nvme0n1.out‚31825.9248046875 disk.dm-1.in‚0 disk.sda.out‚ 51.3359375 disk.sda3.in‚0 disk.drbd1260.out‚0.234375 network.recv‚527200 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 4485.73828125# proc.memory.rssMB‚ 3262.16015625) disk./.usagePercent‚77.31029877553848$ disk./.usageGB‚1151.979320526123 disk.dm-0.in‚ 3484.015625" disk.dm-0.out‚31825.9248046875 disk.dm-1.out‚ 51.3359375 disk.sda.in‚0 disk.sda3.out‚ 51.3359375 disk.drbd1260.in‚0 disk.sda1.in‚0µJ9Srä‚(:ç ÈÐçÔæ’ˆy disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚19.910645484924316 cpu‚24.342278932160287 network.sent‚ 1.526426e+06 disk.dm-0.in‚ 3484.015625 disk.sda3.in‚0 disk.drbd1260.out‚0.234375 disk.nvme0n1.in‚ 3484.015625 disk.sda2.out‚0# proc.memory.rssMB‚ 3262.16015625% disk./.usageGB‚1150.8988418579102( disk.nvme0c0n1.out‚32262.60302734375# disk.dm-0.out‚32262.60302734375 disk.sda2.in‚0# memory_percent‚72.6222276687622 network.recv‚530335! disk.nvme0c0n1.in‚ 3484.015625 disk.dm-1.out‚ 51.76171875 disk.sda.out‚ 51.76171875 disk.sda3.out‚ 51.76171875& disk.nvme0n1.out‚32262.60302734375) proc.memory.availableMB‚ 4485.57421875 proc.cpu.threads‚17) disk./.usagePercent‚77.23778694553273 disk.dm-1.in‚0 disk.sda.in‚0…7ÂŽå‚(:… ÈÐçÔݽ½y gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚37.0 gpu.0.powerWatts‚130.243' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¡^2æ- _wandbruntime‚1582  _runtime‚1582<çÓ-Äç¾ _step‚76 & lr-AdamW/pg1‚0.0005386271688821402 & lr-AdamW/pg2‚0.0005386271688821402 trainer/global_step‚999 $ _timestamp‚1.788471375431774e+09 _runtime‚1582.188014061òfÚ•Èè & lr-AdamW/pg1‚0.0005386271688821402 & lr-AdamW/pg2‚0.0005386271688821402 trainer/global_step‚999 $ _timestamp‚1.788471375431774e+09 _runtime‚1582.188014061 _step‚76Lý—,å2é- _wandbruntime‚1582  _runtime‚1582&¬¦f÷'êñO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150378967701008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139339847428096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3333502808064 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152783307530120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156233848006144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153825313718272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3333573415936 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12806519 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142673299033088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 _runtime‚1582.189576917 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156237484291584 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2407973912216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 _step‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139339796225024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¸‰ïøxëõO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_Í»N»ƒ'start/segment.large_pool.current‚1532 $ _timestamp‚1.788471375433484e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153825287326720 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121810944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153825287326720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24649200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2412170573312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45083901 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142673420844032 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24624358 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8883517 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152786941613224 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20434701 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2408560679424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21690036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12806514 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20434695 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21689334 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚70607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156233848006144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153825313718272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3333573415936 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12806519 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142673299033088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 _runtime‚1582.189576917 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156237484291584 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2407973912216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 _step‚77 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139339796225024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150378967701008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139339847428096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3333502808064 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152783307530120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150529271514176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2410971161088 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 % _timestamp‚1.7884713764614558e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8892342 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3336921745408 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24648925 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21711666 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139479086614016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152936043074542 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2410971161088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153979091277312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1583.219276865 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21710973 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150529296942144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2406771560366 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚78 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45128918 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20455125 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2414581068288 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8891654 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12819324 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20455119 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2410380215710 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20455119 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2410971161088 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 % _timestamp‚1.7884713764614558e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8892342 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3336921745408 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24648925 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21711666 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139479086614016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152936043074542 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2410971161088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153979091277312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1583.219276865 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21710973 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚150529296942144 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2406771560366 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚78 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45128918 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20455125 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2414581068288 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8891654 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12819324 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20455119 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2410380215710 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20455119 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150529271514176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161514191670272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3500303135744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146304315674624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9326163 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21455895 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161514165278720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2529099176448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161514191670272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13446769 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25856060 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164046900740608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9326850 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149804550299648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164043264455168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2532709070336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13446764 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22772927 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884714587305987e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21455901 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21455895 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164043264455168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21455901 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3500234625024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157895429757808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2532709070336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2528302667724 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157895404329840 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47311955 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072É&œý'ƒ÷O O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157895404329840 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47311955 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149804670013440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2524694012588 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 _step‚81 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25880902 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25856060 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161514191670272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3500303135744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146304315674624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9326163 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21455895 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161514165278720 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2529099176448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161514191670272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13446769 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25856060 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164046900740608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚692 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9326850 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149804550299648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164043264455168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2532709070336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13446764 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22772927 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7884714587305987e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21455901 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21455895 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164043264455168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21455901 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3500234625024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157895429757808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2532709070336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2528302667724QðŽ"2„- _wandbruntime‚1666  _runtime‚1666öí¤Y…ýM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25905495 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164203088794624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160576467970162 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47381820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9335101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2535119565312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3503653201920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158045758998944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25880627 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9335760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13459574 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160572833886850 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2531509658112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25880627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22794670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚161667969229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149947190768128 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2531509658112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22795334 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119700480 XJDeviceStatsMonitor.on_traç¨EXªin_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 _runtime‚1666.520542974 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21476319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1049 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47356946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884714597643766e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2535119565312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146443606063616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47381820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161667942837760 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161667942837760 _step‚82 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21476325 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47356946 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21476319 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2527100315874 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21476325 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164203088794624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25905495 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0©(^'†N @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2527100315874 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21476325 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164203088794624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25905495 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25905495 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164203088794624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160576467970162 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47381820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9335101 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2535119565312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚664 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3503653201920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158045758998944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25880627 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9335760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13459574 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160572833886850 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2531509658112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25880627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22794670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚161667969229312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149947190768128 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2531509658112 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22795334 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119700480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 _runtime‚1666.520542974 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21476319 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1049 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47356946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884714597643766e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2535119565312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146443606063616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47381820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161667942837760 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161667942837760 _step‚82 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21476325 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47356946 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21476319 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921R„ù»M2‡- _wandbruntime‚1666  _runtime‚1666<”`PˆÊ 2loss/SRqqt4/assignment_loss‚2.6215827340303464 /loss/FBt2/detection_loss‚1.0275680871183674 /loss/FBt4/detection_loss‚2.9398708164691927 /loss/FRt1/assignment_loss‚4.340687815283165 2loss/SRqqt1/assignment_loss‚2.5248574872966856 trainer/global_step‚1049 /loss/FRt4/assignment_loss‚5.760131966500055 1loss/FBt2/assignment_loss‚0.28320480035331136 0loss/FBt4/assignment_loss‚0.5945257335901261 _runtime‚1666.522787855 /loss/FRt3/assignment_loss‚5.570023088744192 0loss/FBt3/assignment_loss‚0.7246567667461932 ,loss/FRt1/detection_loss‚1.9240355655154 /loss/FRt2/detection_loss‚1.8896679663709526 .loss/FRt4/detection_loss‚2.758718726180849 1loss/SRqqt1/detection_loss‚0.8735831442754716 1loss/SRqqt3/detection_loss‚2.5372093030900666 % _timestamp‚1.7884714597669954e+09 0loss/FBt1/assignment_loss‚0.6621545363838474 epoch‚0 _step‚83 1loss/SRqqt2/assignment_loss‚4.362088521321614 .loss/FBt3/detection_loss‚6.000540614128113 %loss/total_loss‚67.56029815459566 2loss/SRqqt3/assignment_loss‚2.7678382342511956 -loss/FRt3/detection_loss‚2.99853777885437 0loss/SRqqt2/detection_loss‚4.708989540735881 0loss/SRqqt4/detection_loss‚2.324865234452625 -loss/FBt1/detection_loss‚3.60012861341238 0loss/FRt2/assignment_loss‚3.7648310794912536@#œT‰Î 1loss/FBt2/assignment_loss‚0.28320480035331136 0loss/FBt4/assignment_loss‚0.5945257335901261 _runtime‚1666.522787855 /loss/FRt3/assignment_loss‚5.570023088744192 0loss/FBt3/assignment_loss‚0.7246567667461932 ,loss/FRt1/detection_loss‚1.9240355655154 /loss/FRt2/detection_loss‚1.8896679663709526 .loss/FRt4/detection_loss‚2.758718726180849 1loss/SRqqt1/detection_loss‚0.8735831442754716 1loss/SRqqt3/detection_loss‚2.5372093030900666 % _timestamp‚1.7884714597669954e+09 0loss/FBt1/assignment_loss‚0.6621545363838474 epoch‚0 _step‚83 1loss/SRqqt2/assignment_loss‚4.362088521321614 .loss/FBt3/detection_loss‚6.000540614128113 %loss/total_loss‚67.56029815459566 2loss/SRqqt3/assignment_loss‚2.7678382342511956 -loss/FRt3/detection_loss‚2.99853777885437 0loss/SRqqt2/detection_loss‚4.708989540735881 0loss/SRqqt4/detection_loss‚2.324865234452625 -loss/FBt1/detection_loss‚3.60012861341238 0loss/FRt2/assignment_loss‚3.7648310794912536 2loss/SRqqt4/assignment_loss‚2.6215827340303464 /loss/FBt2/detection_loss‚1.0275680871183674 /loss/FBt4/detection_loss‚2.9398708164691927 /loss/FRt1/assignment_loss‚4.340687815283165 2loss/SRqqt1/assignment_loss‚2.5248574872966856 trainer/global_step‚1049 /loss/FRt4/assignment_loss‚5.760131966500055S}I+(sŠ‚(:è ±ÑçÔ®œïy network.recv‚560768 disk.sda3.out‚ 55.19140625 disk.drbd1260.out‚0.25( disk.nvme0c0n1.out‚34024.17529296875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 55.19140625) disk./.usagePercent‚77.28814234748013! disk.nvme0n1.in‚ 3734.01953125 disk.sda2.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 3734.01953125# disk.dm-0.out‚34024.17529296875" memory_percent‚72.621750831604) proc.memory.availableMB‚ 4485.65234375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 3734.01953125# proc.memory.rssMB‚ 3262.16015625% disk./.usageGB‚1151.6491737365723& disk.nvme0n1.out‚34024.17529296875 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 55.19140625* proc.memory.percent‚19.910645484924316 cpu‚24.395075321022905 proc.cpu.threads‚17 network.sent‚ 1.609495e+06ØsÊ`‘‹‚(:† ±ÑçÔ¦ØÀz gpu.0.gpu‚53.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚43.0 gpu.0.powerWatts‚222.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.5634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:¶Ñ·’Œ‚(:‡ ÀÑçÔÍõw gpu.0.gpu‚32.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.368' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ºm‚(:â ÀÑçÔ´…y disk.sda.out‚ 55.859375 network.recv‚565410% disk./.usageGB‚1151.6491355895996# disk.nvme0c0n1.in‚ 3734.01953125( disk.nvme0c0n1.out‚34348.55419921875 disk.dm-1.in‚0 disk.dm-1.out‚ 55.859375 disk.sda1.in‚0) disk./.usagePercent‚77.28813978740479# disk.dm-0.out‚34348.55419921875 disk.drbd1260.out‚0.25! disk.nvme0n1.in‚ 3734.01953125& disk.nvme0n1.out‚34348.55419921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 55.859375 network.sent‚ 1.65831e+06 disk.dm-0.in‚ 3734.01953125 disk.drbd1260.in‚0$ memory_percent‚72.62177467346191( proc.memory.availableMB‚ 4485.6484375# proc.memory.rssMB‚ 3262.16015625 disk.sda.in‚0* proc.memory.percent‚19.910645484924316 cpu‚24.402396103995066 proc.cpu.threads‚17{çå¶vŽ‚(:ë ÏÑçÔíÀ¾x network.recv‚570299' disk.nvme0c0n1.out‚34529.3193359375 disk.dm-0.in‚ 3734.01953125 disk.dm-1.in‚0 disk.dm-1.out‚ 56.48828125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0% disk./.usageGB‚1151.4845848083496# disk.nvme0c0n1.in‚ 3734.01953125" disk.dm-0.out‚34529.3193359375! disk.nvme0n1.in‚ 3734.01953125 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚72.62177467346191) disk./.usagePercent‚77.27709664640781 disk.sda.out‚ 56.48828125 disk.sda3.out‚ 56.48828125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.25390625% disk.nvme0n1.out‚34529.3193359375( proc.memory.availableMB‚ 4485.6484375* proc.memory.percent‚19.910645484924316 network.sent‚ 1.664696e+06# proc.memory.rssMB‚ 3262.16015625 cpu‚24.40931488908569 proc.cpu.threads‚171ßêhœ‚(:‘ ÏÑçÔ½”Ôx gpu.0.gpu‚52.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚209.66' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.931999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ µ‚(:’ ÞÑçÔ²ÔÑw gpu.0.gpu‚52.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚132.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.591199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Œ³á]‘‚(:Ò ÞÑçÔ©À¦x" disk.dm-0.out‚35263.3466796875 disk.sda.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 4485.64453125* proc.memory.percent‚19.910645484924316 cpu‚24.416107143508 disk.sda3.out‚56.9375 network.sent‚ 1.669553e+06) disk./.usagePercent‚77.28593581854251 disk.nvme0c0n1.in‚ 3946.5625 disk.dm-0.in‚ 3946.5625 disk.dm-1.out‚56.9375 disk.sda.out‚56.9375 disk.drbd1260.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.25390625 disk.nvme0n1.in‚ 3946.5625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚72.62179851531982# proc.memory.rssMB‚ 3262.16015625 network.recv‚573444% disk./.usageGB‚1151.6162948608398% disk.nvme0n1.out‚35263.3466796875 disk.sda1.out‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚35263.3466796875Ô`0kk’‚(:à íÑçÔ›î°y# disk.dm-0.out‚35559.66064453125 network.recv‚578064( disk.nvme0c0n1.out‚35580.03564453125 disk.dm-1.in‚0$ memory_percent‚72.62184619903564) proc.memory.availableMB‚ 4485.63671875# proc.memory.rssMB‚ 3262.16015625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 3946.5625 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 3946.5625) disk./.usagePercent‚77.29796049242577% disk./.usageGB‚1151.7954711914062& disk.nvme0n1.out‚35580.03564453125 disk.sda2.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 3946.5625 disk.sda.in‚0 disk.sda.out‚ 57.3984375 network.sent‚ 1.675686e+06 disk.dm-1.out‚ 57.3984375 disk.sda3.in‚0 disk.sda3.out‚ 57.3984375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2578125* proc.memory.percent‚19.910645484924316 cpu‚24.422618218337124ÁÌs“‚(:… íÑçÔ‰×y gpu.0.gpu‚4.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚38.0 gpu.0.powerWatts‚154.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.9882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºµ´2”- _wandbruntime‚1750  _runtime‚1750'EEwÆ•À & lr-AdamW/pg2‚0.0005925438024038759 trainer/global_step‚1099 % _timestamp‚1.7884715439771745e+09 _runtime‚1750.734266047 _step‚84 & lr-AdamW/pg1‚0.0005925438024038759AÍ{?Ê–Ä trainer/global_step‚1099 % _timestamp‚1.7884715439771745e+09 _runtime‚1750.734266047 _step‚84 & lr-AdamW/pg1‚0.0005925438024038759 & lr-AdamW/pg2‚0.0005925438024038759TÎÂþ2—- _wandbruntime‚1750  _runtime‚1750¾Nh’ï˜óO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 trainer/global_step‚1099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2645022768096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171852680904192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚169203043230720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3667102372352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27112604 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2653247567360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3667170883072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22477101 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2649637673472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2648631423232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2653247567360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27112604 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156935937496576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171852680904192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156936057210368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171856317189632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49589705 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168056889154736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14087014 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23856940 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153268886327296 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27087762 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169203069622272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165411866386640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696ŒÊšðý'™÷O [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2653247567360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27112604 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156935937496576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171852680904192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156936057210368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171856317189632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49589705 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168056889154736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14087014 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23856940 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153268886327296 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27087762 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169203069622272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165411866386640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 trainer/global_step‚1099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2645022768096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3636285440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171852680904192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚169203043230720 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3667102372352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27112604 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2653247567360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3667170883072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22477101 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2649637673472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2648631423232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0U<*q¶2š- _wandbruntime‚1753  _runtime‚1753Ïm_Ù'›üM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68497408 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2652048155136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27137197 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27112329 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2655658062336 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23879315 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2651037726726 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22497519 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23878615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14099824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168209624699158 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14099819 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 $ _timestamp‚1.788471546762084e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27137197 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153408176716288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49634722 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22497525 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172012505243648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2655658062336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27112329 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157078674868224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165562221055744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2652048155136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3670429654528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 _runtime‚1753.522691083 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49609848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22497519 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3670498151936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165562195627776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168213258782470 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2647429071382 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22497519 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23878615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14099824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168209624699158 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14099819 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 $ _timestamp‚1.788471546762084e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27137197 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153408176716288 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49634722 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22497525 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172012505243648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2655658062336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27112329 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157078674868224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165562221055744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2652048155136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3670429654528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 _runtime‚1753.522691083 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49609848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22497519 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3670498151936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165562195627776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168213258782470 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2647429071382 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚68497408 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2652048155136 TEDeviceStatsMonito0!Ádr.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27137197 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27112329 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2655658062336 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23879315VÇë¬2- _wandbruntime‚1753  _runtime‚1753Çá5XžÒ /loss/FBt2/detection_loss‚0.7774981713382068 1loss/FBt2/assignment_loss‚0.39992672833009296 .loss/FRt2/detection_loss‚2.439413007526171 0loss/FRt2/assignment_loss‚3.4909967936220623 1loss/SRqqt2/detection_loss‚3.4668102661768594 1loss/SRqqt3/detection_loss‚3.0707603295644126 _step‚87 /loss/FRt3/assignment_loss‚5.552368118212773 2loss/SRqqt1/assignment_loss‚1.9062424432199734 1loss/SRqqt4/assignment_loss‚2.949581594765186 0loss/FBt1/assignment_loss‚0.9028166863653395 /loss/FBt4/detection_loss‚3.0094779913242045 &loss/total_loss‚62.613425051663654 epoch‚0 /loss/FRt1/detection_loss‚1.8812935662240573 0loss/SRqqt1/detection_loss‚0.940695939384974 $ _timestamp‚1.788471546769481e+09 _runtime‚1753.526449187 0loss/FRt1/assignment_loss‚3.3216366811710247 /loss/FRt4/assignment_loss‚5.389375227870363 1loss/FBt3/assignment_loss‚0.44702310860157013 0loss/FBt4/assignment_loss‚0.5365621689869351 /loss/FRt3/detection_loss‚3.2241380398090067 0loss/SRqqt4/detection_loss‚2.276054911315441 2loss/SRqqt3/assignment_loss‚2.5681767747515725 /loss/FRt4/detection_loss‚3.0413296764547173 /loss/FBt1/detection_loss‚3.1212138136227927 .loss/FBt3/detection_loss‚4.804925362269084 trainer/global_step‚1099 1loss/SRqqt2/assignment_loss‚3.095107650756836b#;\ŸÖ 1loss/FBt3/assignment_loss‚0.44702310860157013 0loss/FBt4/assignment_loss‚0.5365621689869351 /loss/FRt3/detection_loss‚3.2241380398090067 0loss/SRqqt4/detection_loss‚2.276054911315441 2loss/SRqqt3/assignment_loss‚2.5681767747515725 /loss/FRt4/detection_loss‚3.0413296764547173 /loss/FBt1/detection_loss‚3.1212138136227927 .loss/FBt3/detection_loss‚4.804925362269084 trainer/global_step‚1099 1loss/SRqqt2/assignment_loss‚3.095107650756836 /loss/FBt2/detection_loss‚0.7774981713382068 1loss/FBt2/assignment_loss‚0.39992672833009296 .loss/FRt2/detection_loss‚2.439413007526171 0loss/FRt2/assignment_loss‚3.4909967936220623 1loss/SRqqt2/detection_loss‚3.4668102661768594 1loss/SRqqt3/detection_loss‚3.0707603295644126 _step‚87 /loss/FRt3/assignment_loss‚5.552368118212773 2loss/SRqqt1/assignment_loss‚1.9062424432199734 1loss/SRqqt4/assignment_loss‚2.949581594765186 0loss/FBt1/assignment_loss‚0.9028166863653395 /loss/FBt4/detection_loss‚3.0094779913242045 &loss/total_loss‚62.613425051663654 epoch‚0 /loss/FRt1/detection_loss‚1.8812935662240573 0loss/SRqqt1/detection_loss‚0.940695939384974 $ _timestamp‚1.788471546769481e+09 _runtime‚1753.526449187 0loss/FRt1/assignment_loss‚3.3216366811710247 /loss/FRt4/assignment_loss‚5.389375227870363W×òŽ'œ ‚(:‘ üÑçÔàõøx gpu.0.gpu‚33.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚37.0 gpu.0.powerWatts‚134.591' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.918200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Î¡f¡‚(:Û üÑçÔŽ—¹‚) disk./.usagePercent‚77.29792439536345% disk./.usageGB‚1151.7949333190918 disk.nvme0n1.in‚ 3946.5625 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚72.62537479400635# proc.memory.rssMB‚ 3262.16015625 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.out‚57.84375 disk.sda.in‚0 disk.sda.out‚57.84375 disk.sda3.out‚57.84375* proc.memory.percent‚19.910645484924316 cpu‚24.429614775541353& disk.nvme0n1.out‚35888.32080078125 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚581310 disk.nvme0c0n1.in‚ 3946.5625( disk.nvme0c0n1.out‚35888.32080078125 disk.dm-0.in‚ 3946.5625# disk.dm-0.out‚35888.32080078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2578125) proc.memory.availableMB‚ 4485.05859375 network.sent‚ 1.680547e+06àIÞ¢‚(:„ ‹ÒçÔùçw gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚46.0 gpu.0.powerWatts‚285.75' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.15 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Í_£‚(:Ô ‹ÒçÔÅ×åz disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 3946.5625' disk.nvme0c0n1.out‚36044.8291015625 disk.sda.out‚58.3125( proc.memory.availableMB‚ 4485.5546875 network.sent‚ 1.730875e+06 disk.sda3.out‚58.3125 disk.nvme0n1.in‚ 3946.5625 disk.dm-0.in‚ 3946.5625# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 cpu‚24.435211334180188 network.recv‚587298) disk./.usagePercent‚77.22640408253781% disk./.usageGB‚1150.7292289733887 disk.sda3.in‚0" disk.dm-0.out‚36044.8291015625 disk.dm-1.out‚58.3125$ memory_percent‚72.62234687805176 proc.cpu.threads‚17 disk.drbd1260.in‚0% disk.nvme0n1.out‚36044.8291015625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd1260.out‚ 0.26171875‘ß÷n¤‚(:’ šÒçÔó¦¬w gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚47.0 gpu.0.powerWatts‚284.793' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.958600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09–t¥‚(:é šÒçÔøýùw* proc.memory.percent‚19.910645484924316 cpu‚24.440322958690096 disk.sda3.in‚0 network.sent‚ 1.735745e+06% disk./.usageGB‚1150.7052612304688' disk.nvme0c0n1.out‚36239.8291015625" disk.dm-0.out‚36239.8291015625 disk.dm-1.out‚ 58.73828125 disk.sda.in‚0 disk.sda2.in‚0) disk./.usagePercent‚77.22479558720018 disk.dm-1.in‚0 disk.sda.out‚ 58.73828125( proc.memory.availableMB‚ 4485.5546875 proc.cpu.threads‚17 network.recv‚590361 disk.dm-0.in‚ 4017.5703125 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚72.62234687805176# proc.memory.rssMB‚ 3262.16015625" disk.nvme0c0n1.in‚ 4017.5703125 disk.sda3.out‚ 58.73828125 disk.drbd1260.out‚ 0.26171875 disk.nvme0n1.in‚ 4017.5703125% disk.nvme0n1.out‚36239.8291015625HŠl’¦‚(:‡ ©ÒçÔŒææw gpu.0.gpu‚71.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚47.0 gpu.0.powerWatts‚268.523' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚®*t§‚(:é ©ÒçÔ탗y network.recv‚594806) disk./.usagePercent‚77.22479993932826' disk.nvme0c0n1.out‚36505.7099609375 proc.cpu.threads‚17 network.sent‚ 1.741838e+06% disk./.usageGB‚1150.7053260803223 disk.dm-1.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 4139.55078125 disk.sda2.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 4485.296875# disk.nvme0c0n1.in‚ 4139.55078125 disk.dm-0.in‚ 4139.55078125" disk.dm-0.out‚36505.7099609375 disk.dm-1.out‚ 59.30078125 disk.sda.in‚0 disk.sda3.out‚ 59.30078125 disk.drbd1260.in‚0 disk.drbd1260.out‚0.265625 disk.sda.out‚ 59.30078125% disk.nvme0n1.out‚36505.7099609375 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚72.62392044067383# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 cpu‚24.445508917088386ÿ™’„2¨- _wandbruntime‚1811  _runtime‚1811è’xÅ©¿ & lr-AdamW/pg2‚0.0006195021191647438 trainer/global_step‚1149 $ _timestamp‚1.788471605154537e+09 _runtime‚1811.910297108 _step‚88 & lr-AdamW/pg1‚0.0006195021191647438+|Úìɪà & lr-AdamW/pg1‚0.0006195021191647438 & lr-AdamW/pg2‚0.0006195021191647438 trainer/global_step‚1149 $ _timestamp‚1.788471605154537e+09 _runtime‚1811.910297108 _step‚88X‹î Î2«- _wandbruntime‚1811  _runtime‚1811%0oú'¬ôO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176891947574272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176891947574272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179665733638656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164066991507456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164067111221248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24942060 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2768960178740 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2765351523604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172928328443440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23498301 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28319464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 _runtime‚ 1811.91141257 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24942744 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179662097353216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10214796 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14727269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51817759 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179662097353216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23498301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2773786064384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51842607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2773786064384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176891921182720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23498295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28344306 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2770176170496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175693679967044 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717öŠ?sþ'­øO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚679 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176891921182720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23498295 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28344306 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2770176170496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507759356 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175693679967044 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176891947574272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176891947574272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179665733638656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164066991507456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164067111221248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24942060 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2768960178740 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2765351523604 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172928328443440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23498301 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28319464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 _runtime‚ 1811.91141257 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24942744 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608655136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179662097353216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10214796 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14727269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51817759 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179662097353216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23498301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119713792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2773786064384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51842607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2773786064384Y·``52®- _wandbruntime‚1813  _runtime‚1813S«‘/ ¯ýM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28368899 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚175850049594778 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51887624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14740074 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51887624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2772586652160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51862750 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164209730028032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28368899 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160372696165888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2772586652160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175846415511466 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177045725133312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14740069 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2767757826890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 K>DeviceStatsMonitor.on_train_batch_end/inactÕ@„üive_split.all.freed‚24963741 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28344031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24964413 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23518725 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2776196559360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160372644962816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177045725133312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 _runtime‚1813.058438163 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23518719 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179821921692672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10224339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28344031 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177045698741760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164209608230400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3837033862144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173078657684576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51862750 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179818285393920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23518719 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 # _timestamp‚1.78847160630158e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10223672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2771366482234 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 _runtime‚1813.058438163 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23518719 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179821921692672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507759356 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10224339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28344031 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177045698741760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164209608230400 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3837033862144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173078657684576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51862750 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179818285393920 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23518719 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 # _timestamp‚1.78847160630158e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10223672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2771366482234 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51887624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14740074 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51887624 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2772586652160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51862750 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164209730028032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28368899 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160372696165888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2772586652160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175846415511466 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚177045725133312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14740069 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2767757826890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24963741 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121797632 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28344031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24964413 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚70594560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23518725 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2776196559360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160372644962816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚177045725133312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868Z/R2±- _wandbruntime‚1813  _runtime‚1813Cè( W²Ñ /loss/FRt4/assignment_loss‚5.355489204669821 /loss/FBt2/detection_loss‚0.7620963416691241 /loss/FRt2/assignment_loss‚3.605188398134141 /loss/FBt3/detection_loss‚3.9466353257497153 _runtime‚1813.060106808 0loss/FBt3/assignment_loss‚0.6188208433256174 1loss/SRqqt3/detection_loss‚2.7479412637907883 1loss/SRqqt4/detection_loss‚2.7657224033027887 2loss/SRqqt1/assignment_loss‚1.8793883487582206 2loss/SRqqt3/assignment_loss‚2.3379833770209344 2loss/SRqqt4/assignment_loss‚2.7511839354410768 0loss/FBt4/assignment_loss‚0.5291226605574164 /loss/FRt3/assignment_loss‚5.219213590621949 1loss/SRqqt2/assignment_loss‚2.301520600914955 0loss/FBt1/assignment_loss‚0.5095286106483804 .loss/FRt4/detection_loss‚3.071138482669304 1loss/SRqqt1/detection_loss‚0.9976427349299193 1loss/SRqqt2/detection_loss‚3.8668834567070007 %loss/total_loss‚60.26214622128653 epoch‚0 .loss/FBt1/detection_loss‚2.719469679726495 .loss/FBt4/detection_loss‚3.593189948134952 1loss/FBt2/assignment_loss‚0.26856303802760906 .loss/FRt1/detection_loss‚1.762916221380389 /loss/FRt3/detection_loss‚3.0641284918785097 0loss/FRt1/assignment_loss‚3.0043428237476046 /loss/FRt2/detection_loss‚2.5840364394798163 trainer/global_step‚1149 % _timestamp‚1.7884716063046296e+09 _step‚91dm"“[³Õ 1loss/SRqqt2/assignment_loss‚2.301520600914955 0loss/FBt1/assignment_loss‚0.5095286106483804 .loss/FRt4/detection_loss‚3.071138482669304 1loss/SRqqt1/detection_loss‚0.9976427349299193 1loss/SRqqt2/detection_loss‚3.8668834567070007 %loss/total_loss‚60.26214622128653 epoch‚0 .loss/FBt1/detection_loss‚2.719469679726495 .loss/FBt4/detection_loss‚3.593189948134952 1loss/FBt2/assignment_loss‚0.26856303802760906 .loss/FRt1/detection_loss‚1.762916221380389 /loss/FRt3/detection_loss‚3.0641284918785097 0loss/FRt1/assignment_loss‚3.0043428237476046 /loss/FRt2/detection_loss‚2.5840364394798163 trainer/global_step‚1149 % _timestamp‚1.7884716063046296e+09 _step‚91 /loss/FRt4/assignment_loss‚5.355489204669821 /loss/FBt2/detection_loss‚0.7620963416691241 /loss/FRt2/assignment_loss‚3.605188398134141 /loss/FBt3/detection_loss‚3.9466353257497153 _runtime‚1813.060106808 0loss/FBt3/assignment_loss‚0.6188208433256174 1loss/SRqqt3/detection_loss‚2.7479412637907883 1loss/SRqqt4/detection_loss‚2.7657224033027887 2loss/SRqqt1/assignment_loss‚1.8793883487582206 2loss/SRqqt3/assignment_loss‚2.3379833770209344 2loss/SRqqt4/assignment_loss‚2.7511839354410768 0loss/FBt4/assignment_loss‚0.5291226605574164 /loss/FRt3/assignment_loss‚5.219213590621949[!‚¾œ´‚(:‘ ¸ÒçÔéìþx gpu.0.gpu‚57.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚46.0 gpu.0.powerWatts‚232.985' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚46.59700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vgvµ‚(:ë ¸ÒçÔàÖõz disk.drbd1260.in‚0 disk.drbd1260.out‚0.265625! disk.nvme0n1.in‚ 4139.55078125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4139.55078125) proc.memory.availableMB‚ 4485.48828125 network.sent‚ 1.746728e+06) disk./.usagePercent‚77.22689228890562 disk.sda3.in‚0 disk.sda3.out‚ 59.93359375% disk.nvme0n1.out‚36706.2099609375 disk.sda1.out‚0' disk.nvme0c0n1.out‚36706.2099609375 disk.dm-0.in‚ 4139.55078125% disk./.usageGB‚1150.7365036010742 disk.sda.in‚0 cpu‚24.450599783054273 proc.cpu.threads‚17 network.recv‚597922 disk.sda.out‚ 59.93359375 disk.sda2.out‚0 disk.sda1.in‚0" disk.dm-0.out‚36706.2099609375$ memory_percent‚72.62275218963623# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 disk.dm-1.in‚0 disk.dm-1.out‚ 59.93359375V–úq¶‚(:æ ÇÒçÔ€Û•z# disk.nvme0n1.out‚36972.16015625 disk.sda2.out‚0 disk.dm-1.out‚ 60.39453125) proc.memory.availableMB‚ 4485.23828125 cpu‚24.457105639245825 disk.sda3.out‚ 60.39453125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 4139.55078125 disk.dm-1.in‚0# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 network.recv‚603871( disk./.usagePercent‚77.2268899848378% disk./.usageGB‚1150.7364692687988 disk.sda3.in‚0% disk.nvme0c0n1.out‚36972.16015625 disk.sda.out‚ 60.39453125$ memory_percent‚72.62427806854248 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.26953125! disk.nvme0n1.in‚ 4139.55078125 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 4139.55078125 disk.dm-0.out‚36972.16015625 disk.sda.in‚0 network.sent‚ 1.798996e+06ãÕ,m‘·‚(:† ÇÒçÔ€×É{ gpu.0.gpu‚44.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^|:i¸‚(:Þ ÖÒçÔ«§ˆx disk.sda.in‚0 disk.drbd1260.out‚ 0.26953125* proc.memory.percent‚19.910645484924316 cpu‚24.46212825342311 network.recv‚607249! disk.nvme0n1.in‚ 4139.55078125 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 60.74609375 disk.sda.out‚ 60.74609375 disk.sda3.in‚0# proc.memory.rssMB‚ 3262.16015625 disk.dm-0.out‚ 37263.109375% disk./.usageGB‚1150.6354904174805 disk.sda2.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 4139.55078125# disk.nvme0c0n1.out‚ 37263.109375 disk.dm-0.in‚ 4139.55078125 disk.dm-1.in‚0! disk.nvme0n1.out‚ 37263.109375 disk.sda3.out‚ 60.74609375 disk.drbd1260.in‚0$ memory_percent‚72.62277603149414' proc.memory.availableMB‚ 4485.484375 proc.cpu.threads‚17 network.sent‚ 1.802131e+06) disk./.usagePercent‚77.22011320939892ê½þ·œ¹‚(:‘ ÖÒçÔ«í×x gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚47.0 gpu.0.powerWatts‚271.271' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.25420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿags›º‚(: åÒçÔ»ê×x gpu.0.gpu‚7.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚40.0 gpu.0.powerWatts‚131.587' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.317399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝP® p»‚(:å åÒçÔ¦÷ôx disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚72.62277603149414* proc.memory.percent‚19.910645484924316 cpu‚24.467129060723497 disk.sda3.out‚ 61.0859375 network.recv‚611933' disk.nvme0c0n1.out‚37525.1943359375" disk.dm-0.out‚37525.1943359375 disk.sda.out‚ 61.0859375 disk.sda3.in‚0' proc.memory.availableMB‚ 4485.484375 proc.cpu.threads‚17# disk./.usageGB‚1150.92289352417% disk.nvme0n1.out‚37525.1943359375 disk.sda2.out‚0# proc.memory.rssMB‚ 3262.16015625# disk.nvme0c0n1.in‚ 4307.55859375 disk.dm-0.in‚ 4307.55859375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2734375! disk.nvme0n1.in‚ 4307.55859375 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.808276e+06) disk./.usagePercent‚77.23940107303612 disk.dm-1.out‚ 61.0859375ä°Ê¡u¼‚(:ê ôÒçÔ¿­Üy% disk.nvme0n1.out‚37646.6865234375 disk.sda1.in‚0 disk.dm-1.out‚ 61.60546875 disk.sda3.out‚ 61.60546875 network.recv‚615445( disk./.usagePercent‚77.2478175767318 disk.dm-1.in‚0 disk.sda.out‚ 61.60546875# proc.memory.rssMB‚ 3262.16015625* proc.memory.percent‚19.910645484924316 proc.cpu.threads‚17 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4307.55859375' disk.nvme0c0n1.out‚37646.6865234375 disk.dm-0.in‚ 4307.55859375" disk.dm-0.out‚37646.6865234375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2734375% disk./.usageGB‚1151.0483055114746! disk.nvme0n1.in‚ 4307.55859375 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda.in‚0$ memory_percent‚72.62187004089355( proc.memory.availableMB‚ 4485.6328125 cpu‚24.473062464711965 network.sent‚ 1.813113e+06Ä nž‘½‚(:† ôÒçÔôÚ•z gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚40.0 gpu.0.powerWatts‚156.308' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.2616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Ï•c2¾- _wandbruntime‚1888  _runtime‚1888o6F‚Ä¿¾ & lr-AdamW/pg2‚0.0006464604359256118 trainer/global_step‚1199 $ _timestamp‚1.788471681382964e+09 _runtime‚ 1888.14003723 _step‚92 & lr-AdamW/pg1‚0.0006464604359256118¦/AÈÀ _step‚92 & lr-AdamW/pg1‚0.0006464604359256118 & lr-AdamW/pg2‚0.0006464604359256118 trainer/global_step‚1199 $ _timestamp‚1.788471681382964e+09 _runtime‚ 1888.14003723\zÅ¥2Á- _wandbruntime‚1888  _runtime‚1888ðj÷ósÂõO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10659573 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187471513802240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167197925226496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4000506022912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2885680279112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171198499760128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚93 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2894324561408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2890714667520 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187471513802240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2889288934248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 TGDeviceStatsMonitor.on_train_batch_start/inactive_sp»›]–ˆlit.large_pool.freed‚15367514 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167197874023424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4000574533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180444790500240 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183334104862456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183330470779352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54070661 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187475150087680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2894324561408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 trainer/global_step‚1199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184580825526272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184580799134720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29576008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _runtime‚1888.144189296 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2890714667520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171198380046336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24519501 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26027087 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532Ú˜;Fÿ'ÃùO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _runtime‚1888.144189296 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3609893888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2890714667520 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171198380046336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24519501 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26027087 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10659573 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3636285440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187471513802240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24842 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167197925226496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880497613 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚68510720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4000506022912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2885680279112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171198499760128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚93 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2894324561408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2890714667520 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634083104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187471513802240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2889288934248 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15367514 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚167197874023424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4000574533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24842 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180444790500240 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3609893888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509020160 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30504 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183334104862456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183330470779352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54070661 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187475150087680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509020160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2894324561408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 trainer/global_step‚1199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184580825526272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184580799134720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29576008]Ç› Å’Ä‚(:‡ ƒÓçÔ”ÂÙy gpu.0.gpu‚56.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚212.113' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.4226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸LU„uÅ‚(:ê ƒÓçÔõÓÝz' disk.nvme0c0n1.out‚37939.6083984375" disk.dm-0.out‚37939.6083984375 network.recv‚620401 disk.drbd1260.in‚0 disk.dm-0.in‚ 4307.55859375# memory_percent‚72.6243257522583# proc.memory.rssMB‚ 3262.16015625 network.sent‚ 1.81955e+06 disk.sda.in‚0% disk.nvme0n1.out‚37939.6083984375) proc.memory.availableMB‚ 4485.23046875* proc.memory.percent‚19.910645484924316 cpu‚24.47865324516176 proc.cpu.threads‚17) disk./.usagePercent‚77.26496752145057 disk.dm-1.out‚ 61.90234375 disk.drbd1260.out‚ 0.27734375! disk.nvme0n1.in‚ 4307.55859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1151.3038520812988 disk.dm-1.in‚0 disk.sda.out‚ 61.90234375 disk.sda3.in‚0 disk.sda3.out‚ 61.90234375# disk.nvme0c0n1.in‚ 4307.55859375+Wnª2Æ- _wandbruntime‚1890  _runtime‚1890¸9*-'ÇþM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896735056384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171341135377408 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2888086582398 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171341015676928 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10669161 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183483206323774 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 % _timestamp‚1.7884716838183084e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184734576693760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54115652 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29600601 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15380324 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180595145169344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119700480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187631338141696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183486840407086 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24539925 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54140526 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167337215615488 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180595119741376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29600601 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184734603085312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2893125149184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29575733 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10668483 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26048802 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184734576693760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896735056384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2891695237742 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24539919²—Ô È‚N EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3636298752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896735056384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509020160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3636298752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24868 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608655344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171341135377408 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634083312 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2888086582398 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171341015676928 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10669161 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880497613 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183483206323774 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509020160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 % _timestamp‚1.7884716838183084e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184734576693760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54115652 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29600601 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15380324 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180595145169344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119700480 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187631338141696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183486840407086 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24539925 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54140526 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167337215615488 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3609907200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180595119741376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29600601 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184734603085312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freeíTDüý d‚ 2893125149184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29575733 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3609907200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10668483 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26048802 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184734576693760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896735056384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2891695237742 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24539919 PDeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPü>ínàbi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JPh6_mábh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JP¯/|lâbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JP“‹÷Ãnãbi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JPzª˜xäbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP:1!qåbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPœ‡pæbk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JPp$•kçbf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JPïsêoèbj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JPR¿|ébw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP}—šqêbl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPpCÚ½lëbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JP¡0hŽiìbd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JPD:7gíbb ?DeviceStatsMonitor.on_validation_batch_start/active.all.current"trainer/global_step2:JP39Ftîbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP‹XÁ4pïbk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JPЋ$~ðby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPÀ‚‚mñbh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPô– )oòbj GDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current"trainer/global_step2:JP²Šºoóbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JP´8Äpôbk HDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated"trainer/global_step2:JP:†ûšnõbi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP Kßyöbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP²Î]t÷bo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPv“ù|øbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPšzµ5pùbk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JPdzï»lúbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JP{œz»qûbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPÿyübt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP·E4Hrýbm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPUï)yþbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP8[8mÿbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JPq˜¨ùv€bq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÆP„ibd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JP¥ƒjÉm‚bh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JPÕSÕcwƒbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPQ#o?u„bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPäÇ}×t…bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPçdt†bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP[B™M~‡by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPb%Ñzˆbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP‚]S‹l‰bg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JPíñåÌtŠbo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP÷hí n‹bi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP–%vŒbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPœe7ªxbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPÒLŒlŽbg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JP÷2¸xobj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JPÀ4ybt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP#ZÏur‘bm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP°"h’bc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JPp ¢rk“bf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JPf_È(u”bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPs5Íhz•bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP»V$u–bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPm ©j—be BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JP­(¹3q˜bl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JP5hre™b` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JPXÄ)všbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPº4Àt›bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP‚)ûërœbm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JP^ĺÿmbh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JPÿÝ]žbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JP|õF qŸbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPS·„t bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPº§‡n¡bi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JP/ñ<²p¢bk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JP­ï×l£bg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPõ1ôVs¤bn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP!ê}„v¥bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPTŽØd¦b_ ìw©br ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP´o˜-vªbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPâVCµq«bl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JPçìäRk¬bf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JP7QÞr­bm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPñóõ m®bh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPP5, k¯bf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JP- ˜jw°br ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP—ãebe±b` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JPËj¡No²bj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JP˜wEl³bg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JPßT™m´bh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JP Î%tµbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPL¸áv¶bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPA{Êu·bp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPé59,f¸ba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP±N¢t¹bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPvs—Ûrºbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP\ò0¤s»bn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JP¤Kût¼bo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPT:Pìw½br ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP‘…st¾bo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP/B”p¿bk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPÚ0ZdÀb_ DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 25168546816 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2942714524160 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚30229505 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55735608 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2050 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚189786440011304 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚30249427 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30504 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 33390854144 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2185 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55735608 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708156928 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2732451328 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4061669110784 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1532 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚183 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚25486181 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚194335480321536 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2945422681088 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚25486181 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 120901120 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2729913820 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1532 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4509020160 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚194338212772864 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2940265940020 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2945422681088 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880497613 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 _runtime‚1998.555659078 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚191392765797376 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚16100324 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11031405 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚173255156307968 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4061548209664 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚30229505 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚30249427 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28808577024 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚16100327 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚25486176 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚194338212772864 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚189789169925124 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3717 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚194335480321536 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2942714524160 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚173255115590656 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚186848880654288 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3717 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2185 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55715681 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚191392790091776 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚27131729 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4507759356 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19922 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 33390854144 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 25168546816 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2942714524160 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚30229505 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55735608 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2050 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚189786440011304 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚30249427 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30504 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 33390854144 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2185 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55735608 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708156928 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2732451328 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4061669110784 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1532 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚183 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚25486181 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚194335480321536 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2945422681088 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚25486181 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 120901120 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2729913820 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1532 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4509020160 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚194338212772864 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2940265940020 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2945422681088 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880497613 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 _runtime‚1998.555659078 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚191392765797376 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚16100324 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11031405 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚173255156307968 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4061548209664 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚30229505 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚30249427 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28808577024 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚16100327 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚25486176 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚194338212772864 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚189789169925124 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3717 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚194335480321536 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2942714524160 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚173255115590656 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚186848880654288 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3717 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2185 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55715681 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚191392790091776 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚27131729 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4507759356 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19922 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 33390854144 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPy•VŽpçbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPÁ# tpèbk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JP¡q¶6xébs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP{þ?ójêbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPÔtê—rëbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP§)ö›qìbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP«¬(`oíbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPÅ|»ckîbf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JPÛ›ñlïbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JPy kðbf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JPÅiîmñbh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JPÍÔç.tòbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPƒûµ¸góbb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP[9 kôbf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JP?Ú«põbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPnÆŠmöbh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JP΋òëz÷bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPМΊmøbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JP$Vkùbf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP­¾ppnúbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPWªèsûbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPÝ£ Èqübl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPÒük/pýbk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPš3Adþb_ ër‚bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPT÷Ž tƒbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPK ?w„br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPÿ>{n…bi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPµ6”»m†bh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPÒ›ˆÚl‡bg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPò sˆbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP™  ¡r‰bm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP¡tWëuŠbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP¿òÃm‹bh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JP¤²LDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP× ­k¯bf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPõ4]j°be BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPTìÍÆk±bf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JP'¥½âr²bm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPŸÑ»”u³bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP1î×2|´bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPXIÕ³iµbd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JPdP›r¶bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPaÔ×r·bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPË%-Öl¸bg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPC¿$Vi¹bd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JPiomömºbh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP–JEo»bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP\au i¼bd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPJ€ êd½b_ DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3717 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚191500402307072 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19922 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4062098787840 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1532 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19922 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚55759898 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30504 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4582277120 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55759898 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2732451328 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2943265102336 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚183 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 120901120 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚2185 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 29392819712 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚30258157 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2943265102336 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1181 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3866051072 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 33390854144 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1208 _runtime‚2000.337194291 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4062219688960 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507759356 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚194443667409408AdFp)ÕêR ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55739971 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55739971 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚30258157 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1532 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1532 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚16112669 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 28880497613 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚25501741 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30504 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3717 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚191500402307072 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19922 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4062098787840 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1532 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚19922 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚55759898 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30504 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4582277120 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55759898 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2732451328 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2943265102336 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚183 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 120901120 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚2185 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 29392819712 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚30258157 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2943265102336 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1181 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3866051072 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 33390854144 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1208 _runtime‚2000.337194291 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4062219688960 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507759356 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚194443667409408 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚194446399860736 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4509020160 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚19927 BöyJ­ bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPö¯7DH® bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÉþ²‰J¯ bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPxß%žH° bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPO› J± bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÀv @² b; particle/accuracy_5_of_7"trainer/global_step2:JP—êJ'J³ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP4ËìH´ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPH™0NJµ bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÃu»jH¶ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÅËæJ· bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP«¿]Æ@¸ b; particle/accuracy_2_of_2"trainer/global_step2:JPÜdN¹ bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPO/ÁÇNº bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP¿îN» bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPr„?J¼ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP³°ÍœN½ bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP«‚7N¾ bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP/|VJ¿ bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPðÁb©JÀ bE "Purity/3FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPíà†NÁ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¡uåæJ bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPŒˆ0âNà bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP ÷JÄ bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPå}ÃNÅ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP¨‰HÆ bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPw|xHÇ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPŘö™JÈ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPßκCJÉ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP´é’×NÊ bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPfØPË bI s=j¯H&Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP'õƒCNÌ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP&y­‹NÍ bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPŸóþNÎ bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPØ9øNÏ bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP¶¸ÞNÐ bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP ßÑJÑ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP÷e†1HÒ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPrN ’NÓ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP4§æ@Ô b; particle/accuracy_3_of_3"trainer/global_step2:JPþ™QÿJÕ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP­^¹ÍJÖ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP7féJ× bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPHýYïHØ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP²€ÎgJÙ bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP%r]»NÚ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP*vê NÛ bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPò¾!NÜ bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP$'Š5HÝ bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP¬Àƒ@Þ b; particle/accuracy_5_of_5"trainer/global_step2:JP½úHß bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPEâœiJà bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP{BJJá bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPËuhNâ bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPf6€•Nã bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPFÁ«WHä bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP­FoqJå bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPµ·° Jæ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP«¨©Nç bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP˜ƒÆNè bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPü^¿;é b6 jet/accuracy_3_of_3"trainer/global_step2:JPábg Jê bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPO 4Hë bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP™ [#Nì bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP!á*>Hí bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP®k®ÃHî bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP6ûNï bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP´ìÊHð bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPh¾ÔÑJñ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¿á*YNò bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPq;Hó bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP8ÉÚeJô bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÛc°²Hõ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP;Q}Jö bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPùË¥6N÷ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP•m?Jø bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP6@R%Hù bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPDŠaKHú bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP)ŠˆÜHû bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP—RNü bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP7¼äÃNý bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPѧ-Hþ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP·ùIAJÿ bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP¼Ýs±N€ bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPžcLN bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPàÖ J‚ bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÃ)AÝJƒ bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPý4sµJ„ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP¿ãVËN… bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÌHS¥;† b6 jet/accuracy_1_of_7"trainer/global_step2:JPnù0H‡ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPHLKNˆ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP~H‰ bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP‘º WNŠ bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPȱÿH‹ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPI &ÂHŒ bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPQ€–N bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPX‹NŽ bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP._©!H bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPº»åÓJ bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPXØjJ‘ bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPʦ›mJ’ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPm˼çJ“ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPbßAH… bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP3¶jˆJ† bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPš_ H‡ bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPµÃÖ¨Jˆ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPr¦J‰ bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPÖÇÖ!HŠ bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP lñCN‹ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP ädûNŒ bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÌÞG N bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPœDtJŽ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP7¥LóH bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP3_1•N bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP9‰V¾J‘ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP>I>ÑH’ bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP¢euH“ bC Purity/3FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¿©m”J” bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP› zN• bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP½)ŸJ– bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPp àJ— bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPƒ‡dYN˜ bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPy—)2J™ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPzHÌ bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¼ŽÙ NÍ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPsôINÎ bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÁ¥JœJÏ bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP"µâ[JÐ bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÂæ08JÑ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP>¥¢JÒ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP_ÄÁNÓ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP­”þTHÔ bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPýãÙµHÕ bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP: äNÖ bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP c J× bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPµQcôJØ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPåw?JÙ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPú':÷JÚ bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPH±¡ýJÛ bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPóuJÜ bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP†È£^HÝ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPsm©JÞ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP‹’3(Hß bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÚI¹ÉJà bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPm.s%Há bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPÌŸ<Jâ bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP§†DV;ã b6 jet/accuracy_3_of_6"trainer/global_step2:JPÚ«™ùHä bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPa7JJå bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP^&¢Næ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP™| ÖNç bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÆÛÊNè bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¸)ó;ê b6 jet/accuracy_1_of_4"trainer/global_step2:JPxâ‡èHë bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JP?,Hì bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP,Ž^Jí bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPœT_Jî bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¼ÊNï bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÞ•x]Nð bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP2È@ñ b; particle/accuracy_1_of_5"trainer/global_step2:JP95^Jò bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPù/CJJó bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP7†‹Jô bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÈ*Nõ bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPõG½ê N“ bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP¥¹hª;” b6 jet/accuracy_2_of_4"trainer/global_step2:JPóƒ^¿H• bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP_Ô9B@– b; particle/accuracy_1_of_6"trainer/global_step2:JPLÀ¶ˆ;— b6 loss/val_total_loss"trainer/global_step2:JPK(/J˜ bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPŒiæJ™ bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP:ÊIHš bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP1™ïN› bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPq¦BNœ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPó¿N bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPšÎpHž bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPËÙjöHŸ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPSTJ  bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPol¦N¡ bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP5ÃN¢ bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPΫH£ bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP&¾—ÊN¤ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP˜} ØJ¥ bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPð6nÉH¦ bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPxá¨N§ bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP( YLH¨ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP)XûØH© bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP5wöøNª bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPŸæìH« bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP{€·oJ¬ bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPþ¥wJ­ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPá#íH® bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPF˜~@J¯ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPK"/N° bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPÚ…æH± bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPˆŸŠN² bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPË? eH³ bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPó)¹H´ bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP¿Hµ bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPƒŠÏ{J¶ bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPîÝÀJH· bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPG–²H¸ bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPxðôÎN¹ bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP0ãòHº bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP 8|ÚN» bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPþË›H¼ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP;J½ bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPr¶«hH¾ bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPså üH¿ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP&„©ÀÞ8HÁ bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP:‰3UH bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÏYhHà bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP"’Ž9HÄ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP*JEØNÅ bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPR¢RžNÆ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP$®ö\NÇ bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPÝUÊ=JÈ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÕ ´ÍHÉ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP¾˜S'JÊ bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‚·Ö²HË bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP“èÐ/NÌ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPŽ¹o(NÍ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP¿Šß@Î b; particle/accuracy_1_of_2"trainer/global_step2:JP çóHÏ bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPµyÑ£HÐ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP½ô?‘HÑ bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPrÔö6NÒ bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP½%ÓÞNÓ bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPYõHJÔ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¤©’kHÕ bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÅŠà¢NÖ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPDP„;× b6 jet/accuracy_1_of_5"trainer/global_step2:JPˆö =JØ bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP¸ JÙ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPYÀècNÚ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÎá6rNÛ bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP•^]’HÜ bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP&ÆÚNJÝ bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP*?)ÃJÞ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPd×ëNß bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPûý¼uà bE "Purity/*FRt*SRqqt*FBt/SRÅR¢+qqt_purity"trainer/global_step2:JP‚ëÛ0Há bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPé(Jâ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPâ× ÄJã bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¿~˜ÒNä bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPp‡Ê‘Nå bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPÝ¡¨;æ b6 jet/accuracy_1_of_2"trainer/global_step2:JPíð)‘Hç bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPE0–Nè bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPŠ]øËNé bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP`[€Nê bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPßs.ßJë bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPM¼äHì bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP§Ë„ì;í b6 jet/accuracy_6_of_6"trainer/global_step2:JPÌí’)Hî bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP ÞJ~Jï bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP“¸jHð bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPô ÃŽNñ bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP{3=Nò bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP“H<ó b7 particle/sensitivity"trainer/global_step2:JPBy¥Jô bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPó4ѾJõ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPÃ$YÔHö bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP€^H÷ bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPa³Nø bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPì>úï;ù b6 jet/accuracy_1_of_3"trainer/global_step2:JP—ºà`;ú b6 jet/accuracy_4_of_4"trainer/global_step2:JP°TB¡Jû bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPõÿì³Hü bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPG÷Hý bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÿk¯gNþ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPOä™ÝNÿ bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP¤ByJ€ bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¨r6×J bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPþáJ‚ bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP±¦©Nƒ bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÊ—áÒJ„ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPPn^T;… b6 jet/accuracy_2_of_2"trainer/global_step2:JP9ĸPH† bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPò‰ËJ‡ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPKÄ€Jˆ bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPB­¢N‰ bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPù‹¼JŠ bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPžXÀJ‹ bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÒŠG8Œ b3 particle/f_score"trainer/global_step2:JP–ft¥J bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP‰a<ØNŽ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPQcøÎJ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPV´üJ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP7ù·˜N‘ bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP_§ÇN’ bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPOd¹;“ b6 jet/accuracy_2_of_5"trainer/global_step2:JP‰ÕŠg@” b; particle/accuracy_6_of_6"trainer/global_step2:JPŠNEhN• bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP°2Š"J– bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP´6IkJ— bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP¬0õ’J˜ bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPñ–F´J™ bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPUcÄ&Jš bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP;êêN› bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÒA^WJœ bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPIN bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPƤÚNž bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP$0ÖªNŸ bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP­*ž9  b4 particle/accuracy"trainer/global_step2:JPÍÕ_H¡ bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPy¨ŒJ¢ bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP&¦¾N£ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPW)N;¤ b6 jet/accuracy_1_of_1"trainer/global_step2:JPW@E}@¥ b; particle/accuracy_2_of_3"trainer/global_step2:JP¼’ËH¦ bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP¡${J§ bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÜbH¨ bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPSêJ© bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP­u<;ª b6 jet/accuracy_2_of_6"trainer/global_step2:JP³> >J« bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP™)¾{J¬ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP;„?H­ bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP 1°*J® bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP˜+ J¯ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPôTÙJ° bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP3@± b; particle/accuracy_3_of_7"trainer/global_step2:JP Ê\H² bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPŽ~zìH³ bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPZŸÀyJ´ bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPãéµ;Nµ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÏj©–;¶ b6 jet/accuracy_5_of_7"trainer/global_step2:JPÎU¥N· bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÊEÃJ¸ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPä‚c#H¹ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPZ[nNº bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP]ÑZJ» bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP¡¤yH¼ bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¾~„H½ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP†ß tJ¾ bE "Purity/3FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPª†ôJ¿ bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÞ0”HÀ bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP¹oèNÁ bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPïûqJ bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÞONHà bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP‘™¿JÄ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPF²È£HÅ bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPQþNÆ bI &Purity/1FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP7Z$HÇ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP¹~eJÈ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP}"JÉ bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŸÜ]²HÊ bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP˜‘eLJË bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP•‘ÛHÌ bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPœ¾NÍ bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPñß-€JÎ bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP C'NÏ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPo ¡JHÐ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP*"2cJÑ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÄc8VNÒ bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP¬‡’0NÓ bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPþ Ï4HÔ bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP?>/ÏHÕ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP {NÞJÖ bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÓ©ûxH× bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP·;{JØ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP„o¤kJÙ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPJ—EJÚ bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPz|zHÛ bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÁ/°JÜ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÁb%KNÝ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP&dvNÞ bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP—™ Hß bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPOÞŠ;Jà bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÈëhíJá bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP †#Nâ bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPhäÎSHã bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPyÇû¶@ä b; particle/accuracy_2_of_6"trainer/global_step2:JP+‡Hå bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPáä?JHæ bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP +Hç bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPi(òJè bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÅ&qJé bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP2NÂNê bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP7ÞßeJë bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPFøJì bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPòó¥Hí bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP‡ ¶Nî bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÍfð}Nï bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP˜y—Nð bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPX(üoHñ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP†‚Nò bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPpïa<ó b7 particle/specificity"trainer/global_step2:JPE‰@àHô bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPµÍìjHõ bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPdýÅñHö bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP¿° H÷ bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPªGQJø bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÍx+ðNù bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÝ¥I—Jú bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP\L§Nû bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPI6DNü bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP¬ØHý bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPµïë®Hþ bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPé¢ eHÿ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPòÒ©N€bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP Ù„ÞJbE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÛüuH‚bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPV?žÍJƒbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP½µÂCN„bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÇ]H…bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPí&ÑÓH†bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP«”ÓsJ‡bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¼¢NˆbI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP¯c§˜@‰b; particle/accuracy_4_of_4"trainer/global_step2:JPºÀ8)JŠbE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPòt\±J‹bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP™¸J,JŒbE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPˆtdHbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPȹQMJŽbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPú“цHbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPE.õÕNbI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP®bæN‘bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPØ´rH’bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP7È™ÑN“bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPß¹—ÇN”bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPöïíFJ•bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP$–+vN–bI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPü>(äH—bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÏdÊeH˜bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP‘c+þH™bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPú3‘F@šb; particle/accuracy_4_of_5"trainer/global_step2:JP¨ÆÒH›bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP7ï+8JœbE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPÅ sJbE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP›©HžbC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPóþð÷NŸbI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP¤·ÐÖH bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP±·ïN¡bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPg’|Ï@¢b; particle/accuracy_2_of_5"trainer/global_step2:JPuÌÀ H£bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¡øœÀ;¤b6 jet/accuracy_7_of_7"trainer/global_step2:JPƒ)N¥bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPóñ×N¦bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPPr®2§- _wandbruntime‚2023  _runtime‚2023OU ¡A¨ë¤ 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4053366482257843 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9809389114379883 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.10471128672361374 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2971700131893158 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 jet/accuracy_6_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18120278418064117 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40338578820228577 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_3_of_6‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17611241340637207 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13040277361869812 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9852986931800842 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957460761070251 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1927139312028885 ,jet/accuracy_4_of_5‚0.018817203119397163 6validation_average_jet_accuracy‚0.3767531216144562 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2447560876607895 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13531562685966492 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.37365245819091797 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3407093286514282 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 particle/accuracy_2_of_7‚1 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.38028472661972046 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 jet/accuracy_4_of_7‚0 particle/accuracy_4_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13812515139579773 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666865348816 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18369419872760773 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.20536060631275177 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17418332397937775 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.03787878900766373 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17205475270748138 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3664948642253876 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15059645473957062 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.40086206793785095 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 _runtime‚2023.472143695 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20019662380218506 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6818781495094299 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5522435903549194 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11153628677129745 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3392857015132904 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.493408739566803 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4516991078853607 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22148708999156952 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_1_of_3‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.007566321175545454 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9922016263008118 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16930457949638367 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_4‚0.060068003833293915 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2620437741279602 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17646567523479462 jet/accuracy_2_of_7‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.20922847092151642 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2728724479675293 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3888888955116272 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008127989247441292 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.40509259700775146 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13827790319919586 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1458333283662796 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.13461539149284363 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8264846801757812 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20670537650585175 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.21414390206336975 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5788194537162781 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.34800955653190613 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9890525937080383 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2629706859588623 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9812885522842407 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07173918932676315 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9888132810592651 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975014925003052 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.013392857275903225 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219858795404434 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.29022395610809326 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 )loss/val_total_loss‚58.40876145363735 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.028513828292489052 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.44590383768081665 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722350239753723 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.31214216351509094 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9845208525657654 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3536878228187561 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11398466676473618 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.44988223910331726 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08744855970144272 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994226336479187 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.21992336213588715 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.995634913444519 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9870171546936035 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9873928427696228 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.30043861269950867 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938737750053406 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.503797173500061 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9705460071563721 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_2‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24814820289611816 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9113757610321045 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5101515650749207 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2879464328289032 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8909626603126526 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3736686408519745 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.35783082246780396 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9824930429458618 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4484127163887024 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.668926477432251 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17738083004951477 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3128753900527954 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1492714136838913 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.38690444827079773 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9968820810317993 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7869501113891602 -jet/accuracy_4_of_4‚0.0009920635493472219 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.38876909017562866 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.09732796996831894 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18529415130615234 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.42828795313835144 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5909090638160706 'particle/f_score‚0.4047853946685791 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.28517264127731323 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.43269973993={($ÑP30139 particle/accuracy_6_of_6‚1 epoch‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3858984112739563 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8224309086799622 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08733781427145004 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_1‚0.4040924310684204 particle/accuracy_2_of_3‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.18518519401550293 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5128205418586731 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.36000001430511475 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.39619016647338867 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 trainer/global_step‚1208 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4604937732219696 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_7‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3454407751560211 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08318359404802322 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.017772285267710686 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1194532960653305 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006613756995648146 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2311508059501648 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.989757776260376 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010515784844756126 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21810343861579895 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8993055820465088 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008503400720655918 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.21148021519184113 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3190951645374298 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.34800955653190613 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.018050840124487877 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8144839406013489 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2972048819065094 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0026455027982592583 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993098378181458 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9933072924613953 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +particle/specificity‚0.9030138254165649 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666507720947 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9089834094047546 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4142046868801117 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9871960282325745 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4049702286720276 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5383192300796509 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4581316411495209 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.004629630129784346 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3123376667499542 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2925169765949249 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.46269166469573975 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3350487947463989 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.28875967860221863 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12151099741458893 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34951502084732056 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15840889513492584 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2137034833431244 particle/accuracy_1_of_7‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14290614426136017 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03407225012779236 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32370686531066895 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28787878155708313 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.30067160725593567 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_5‚0.0938556119799614 *jet/accuracy_1_of_6‚0.9679487943649292 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3703342378139496 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.046771686524152756 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.003427128540351987 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1388888955116272 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3713051974773407 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2361110895872116 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.10471128672361374 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33980581164360046 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32276788353919983 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 particle/accuracy_4_of_6‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007724071387201548 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13557764887809753 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18518519401550293 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18710650503635406 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11788910627365112 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5125448107719421 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35651466250419617 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.31192731857299805 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17474746704101562 particle/accuracy_3_of_4‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17274774610996246 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.40471044182777405 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1855120062828064 particle/accuracy_5_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10599789023399353 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.374102383852005 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3586806058883667 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22192195057868958 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07111110538244247 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 % _timestamp‚1.7884718167121327e+09 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18731042742729187 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 -"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3125 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3933333158493042 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2768704891204834 particle/accuracy_5_of_5‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.259855180978775 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 _step‚98 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,jet/accuracy_3_of_3‚0.023166192695498466 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.32935234904289246 4 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1855458766222 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2966221570968628 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.47113093733787537 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1578361541032791 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4085484743118286 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20402298867702484 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 jet/accuracy_1_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09939046204090118 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6432841420173645 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18391813337802887 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.32324329018592834 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1827034056186676 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3411097526550293 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3818311095237732 particle/accuracy_1_of_1‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.40516215562820435 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15137018263339996 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0015069226501509547 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2345118373632431 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48236414790153503 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2732182443141937 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12784884870052338 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.13518518209457397 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14321057498455048 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0193447507917881 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5757575631141663 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18361814320087433 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5#tG!/©ï¤ @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1827034056186676 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3411097526550293 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3818311095237732 particle/accuracy_1_of_1‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.40516215562820435 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15137018263339996 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0015069226501509547 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2345118373632431 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.48236414790153503 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2732182443141937 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12784884870052338 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.13518518209457397 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14321057498455048 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0193447507917881 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5757575631141663 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18361814320087433 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4053366482257843 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9809389114379883 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.10471128672361374 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.2971700131893158 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 jet/accuracy_6_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18120278418064117 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40338578820228577 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 particle/accuracy_3_of_6‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17611241340637207 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13040277361869812 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9852986931800842 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9957460761070251 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1927139312028885 ,jet/accuracy_4_of_5‚0.018817203119397163 6validation_average_jet_accuracy‚0.3767531216144562 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2447560876607895 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13531562685966492 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.37365245819091797 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3407093286514282 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 particle/accuracy_2_of_7‚1 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.38028472661972046 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 jet/accuracy_4_of_7‚0 particle/accuracy_4_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13812515139579773 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666865348816 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18369419872760773 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.20536060631275177 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17418332397937775 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.03787878900766373 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17205475270748138 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3664948642253876 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15059645473957062 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.40086206793785095 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 _runtime‚2023.472143695 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20019662380218506 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6818781495094299 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5522435903549194 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11153628677129745 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3392857015132904 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.493408739566803 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4516991078853607 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22148708999156952 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_1_of_3‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.007566321175545454 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9922016263008118 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16930457949638367 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_4‚0.060068003833293915 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2620437741279602 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17646567523479462 jet/accuracy_2_of_7‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.20922847092151642 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5350877642631531 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2728724479675293 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3888888955116272 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008127989247441292 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.40509259700775146 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13827790319919586 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1458333283662796 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.13461539149284363 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8264846801757812 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.20670537650585175 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.21414390206336975 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5788194537162781 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.34800955653190613 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9890525937080383 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FS·–UcRt1SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2629706859588623 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9812885522842407 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07173918932676315 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9888132810592651 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9975014925003052 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.013392857275903225 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219858795404434 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.29022395610809326 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 )loss/val_total_loss‚58.40876145363735 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.028513828292489052 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.44590383768081665 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722350239753723 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.31214216351509094 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9845208525657654 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3536878228187561 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9883720874786377 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.11398466676473618 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.44988223910331726 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08744855970144272 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994226336479187 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.21992336213588715 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.995634913444519 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9870171546936035 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9873928427696228 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.30043861269950867 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938737750053406 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.503797173500061 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9705460071563721 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_1_of_2‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24814820289611816 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9113757610321045 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5101515650749207 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2879464328289032 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.8909626603126526 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3736686408519745 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.35783082246780396 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9824930429458618 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.4484127163887024 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.668926477432251 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17738083004951477 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3128753900527954 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1492714136838913 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,particle/sensitivity‚0.38690444827079773 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9968820810317993 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7869501113891602 -jet/accuracy_4_of_4‚0.0009920635493472219 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.38876909017562866 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.09732796996831894 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.18529415130615234 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.42828795313835144 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5909090638160706 'particle/f_score‚0.4047853946685791 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.28517264127731323 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4326997399330139 particle/accuracy_6_of_6‚1 epoch‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3858984112739563 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8224309086799622 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08733781427145004 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_1‚0.4040924310684204 particle/accuracy_2_of_3‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.18518519401550293 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5128205418586731 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.36000001430511475 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.39619016647338867 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 trainer/global_step‚1208 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4604937732219696 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_7‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3454407751560211 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.08318359404802322 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.017772285267710686 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1194532960653305 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006613756995648146 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2311508059501648 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.989757776260376 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010515784844756126 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21810343861579895 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8993055820465088 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.008503400720655918 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.21148021519184113 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9855191111564636 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3190951645374298 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.34800955653190613 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.018050840124487877 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8144839406013489 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2972048819065094 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0026455027982592583 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9993098378181458 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9933072924613953 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 +particle/specificity‚0.9030138254165649 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666507720947 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9089834094047546 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4142046868801117 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9871960282325745 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4049702286720276 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5383192300796509 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4581316411495209 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.004629630129784346 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3123376667499542 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2925169765949249 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.46269166469573975 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3350487947463989 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.28875967860221863 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.12151099741458893 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34951502084732056 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_7_of_7‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15840889513492584 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2137034833431244 particle/accuracy_1_of_7‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14290614426136017 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03407225012779236 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32370686531066895 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28787878155708313 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.30067160725593567 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_5‚0.0938556119799614 *jet/accuracy_1_of_6‚0.9679487943649292 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3703342378139496 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.017924528568983078 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.046771686524152756 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.003427128540351987 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1388888955116272 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3713051974773407 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2361110895872116 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.10471128672361374 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33980581164360046 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32276788353919983 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_6‚0 particle/accuracy_4_of_6‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007724071387201548 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13557764887809753 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18518519401550293 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18710650503635406 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11788910627365112 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5125448107719421 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35651466250419617 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.31192731857299805 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17474746704101562 particle/accuracy_3_of_4‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17274774610996246 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.40471044182777405 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1855120062828064 particle/accuracy_5_of_7‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10599789023399353 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.374102383852005 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3586806058883667 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.22192195057868958 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07111110538244247 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 % _timestamp‚1.7884718167121327e+09 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18731042742729187 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 -"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3125 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3933333158493042 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2768704891204834 particle/accuracy_5_of_5‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.259855180978775 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 _step‚98 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,jet/accuracy_3_of_3‚0.023166192695498466 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.32935234904289246 4 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1855458766222 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.2966221570968628 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.47113093733787537 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1578361541032791 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4085484743118286 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.20402298867702484 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 jet/accuracy_1_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09939046204090118 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6432841420173645 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18391813337802887 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.32324329018592834 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0bNÝ›iª‚(:Þ ŠÔçÔíljy disk.dm-1.out‚ 65.6484375 disk.sda.out‚ 65.6484375 disk.sda3.out‚ 65.6484375* proc.memory.percent‚23.981451988220215 cpu‚24.680422410635238 network.recv‚659208 disk.sda1.in‚0# disk.dm-0.out‚41129.24169921875 proc.cpu.threads‚17 network.sent‚ 1.999795e+06& disk.nvme0n1.out‚41129.24169921875 disk.sda2.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 4598.78125( disk.nvme0c0n1.out‚41129.24169921875$ memory_percent‚76.70631408691406# proc.memory.rssMB‚ 3929.12109375% disk./.usageGB‚1152.8084144592285 disk.drbd1260.in‚0 disk.drbd1260.out‚0.296875 disk.sda2.in‚0 disk.dm-0.in‚ 4598.78125 disk.sda.in‚0 disk.sda3.in‚0% proc.memory.availableMB‚ 3816.4375) disk./.usagePercent‚77.36593996504536 disk.nvme0n1.in‚ 4598.78125 disk.dm-1.in‚0ð[ÁA«‚(:„ ŠÔçÔ¡¦Íy gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚33.0 gpu.0.powerWatts‚94.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8978 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çiàf¬‚(:Û ™ÔçÔöÁŠy network.recv‚666880( disk./.usagePercent‚77.7682471008478# disk.dm-0.out‚47679.60791015625 disk.dm-1.out‚66.0625 disk.drbd1260.out‚ 0.30078125% disk./.usageGB‚1158.8030815124512( disk.nvme0c0n1.out‚47694.35791015625 disk.dm-1.in‚0 disk.sda.out‚66.0625 disk.sda3.out‚66.0625 disk.nvme0n1.in‚ 4746.3671875& disk.nvme0n1.out‚47694.60791015625 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 4746.3671875 disk.dm-0.in‚ 4746.3671875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚99.9727725982666% proc.memory.availableMB‚ 4.4609375* proc.memory.percent‚36.389946937561035 network.sent‚ 2.125132e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 5962.12890625 cpu‚24.681964361437828 proc.cpu.threads‚17“ý¨Ž­‚(:ƒ ™ÔçÔ›ÓÄy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.345' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.669 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eã $‘®‚(:† ¨ÔçÔ Ëz gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚122.122' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òaÄw¯‚(:ì ¨ÔçÔ¬ÍÑ| disk.dm-1.in‚0 disk.dm-1.out‚ 66.48046875 disk.sda.out‚ 66.48046875 disk.sda2.in‚0 disk.sda3.out‚ 66.48046875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.30078125 cpu‚24.685491918987168 proc.cpu.threads‚17 network.recv‚670167% disk./.usageGB‚1157.8031616210938& disk.nvme0n1.out‚50482.97900390625 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 4779.05078125 disk.sda.in‚0$ memory_percent‚68.22412014007568* proc.memory.percent‚20.515823364257812 network.sent‚ 2.129945e+06) disk./.usagePercent‚77.70114163795013( disk.nvme0c0n1.out‚50482.97900390625 disk.dm-0.in‚ 4779.05078125 disk.sda3.in‚0) proc.memory.availableMB‚ 5206.16015625 proc.memory.rssMB‚ 3361.3125! disk.nvme0n1.in‚ 4779.05078125 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚50482.97900390625ù"ßØo°‚(:ä ·ÔçÔèžy network.recv‚675340) disk./.usagePercent‚77.69955951138853 disk.sda3.in‚0 disk.sda3.out‚ 67.046875 disk.sda1.out‚0 disk.dm-0.in‚ 4779.05078125% disk./.usageGB‚1157.7795867919922 disk.drbd1260.out‚ 0.3046875 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚68.22600364685059( proc.memory.availableMB‚ 5205.8515625 cpu‚24.689135693905904! disk.nvme0n1.in‚ 4779.05078125& disk.nvme0n1.out‚50639.24267578125 disk.sda.in‚0 disk.sda.out‚ 67.046875 disk.drbd1260.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 4779.05078125( disk.nvme0c0n1.out‚50639.24267578125# disk.dm-0.out‚50639.24267578125 disk.dm-1.out‚ 67.046875 proc.memory.rssMB‚ 3361.3125* proc.memory.percent‚20.515823364257812 proc.cpu.threads‚17 disk.sda2.in‚0 network.sent‚ 2.136244e+067D^’±‚(:‡ ·ÔçÔðâz gpu.0.gpu‚55.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚39.0 gpu.0.powerWatts‚199.807' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.9614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õTz˜u²‚(:ê ÆÔçÔ¬Œ–y# disk.nvme0c0n1.in‚ 4779.05078125 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 4779.05078125# disk.dm-0.out‚51092.18017578125 disk.drbd1260.out‚ 0.3046875& disk.nvme0n1.out‚51092.18017578125 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚68.22636127471924) proc.memory.availableMB‚ 5205.79296875 cpu‚24.69299431583989 disk.dm-1.out‚ 67.36328125 disk.sda.out‚ 67.36328125 network.sent‚ 2.141182e+06) disk./.usagePercent‚77.70113446973917( disk.nvme0c0n1.out‚51092.18017578125 disk.dm-0.in‚ 4779.05078125 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 3361.3125* proc.memory.percent‚20.515823364257812 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 67.36328125 network.recv‚678577% disk./.usageGB‚1157.8030548095703)„œ³‚(:‘ ÆÔçÔìÅ«z gpu.0.gpu‚34.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚37.0 gpu.0.powerWatts‚154.427' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.885399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª@¹Q´‚(:ë ÕÔçÔËɺy* proc.memory.percent‚20.515823364257812) disk./.usagePercent‚77.70113805384466 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 4779.05078125 disk.sda3.out‚ 67.83203125 cpu‚24.696677819345158 proc.cpu.threads‚17 disk.dm-0.in‚ 4779.05078125 network.sent‚ 2.147395e+06 network.recv‚683300 disk.sda2.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚51535.87939453125 disk.dm-1.in‚0 disk.sda.out‚ 67.83203125& disk.nvme0n1.out‚51535.87939453125# disk.dm-0.out‚51535.87939453125 disk.sda3.in‚0 disk.drbd1260.in‚0$ memory_percent‚68.23432445526123) proc.memory.availableMB‚ 5204.48828125$ disk./.usageGB‚1157.803108215332! disk.nvme0n1.in‚ 4779.05078125 disk.sda1.in‚0 disk.dm-1.out‚ 67.83203125 disk.sda.in‚0 disk.drbd1260.out‚ 0.3085*‚Óä%9375 proc.memory.rssMB‚ 3361.3125ÑÖýµ‚(:… ÕÔçÔ’ôÏz gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.157' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ®MÜ2¶- _wandbruntime‚2108  _runtime‚2108ï@ïGÆ·À & lr-AdamW/pg1‚0.0006518303639418193 & lr-AdamW/pg2‚0.0006518303639418193 trainer/global_step‚1249 % _timestamp‚1.7884719018489525e+09 _runtime‚2108.605619996 _step‚99 ê%Ê¸Ä _step‚99 & lr-AdamW/pg1‚0.0006518303639418193 & lr-AdamW/pg2‚0.0006518303639418193 trainer/global_step‚1249 % _timestamp‚1.7884719018489525e+09 _runtime‚2108.605619996c#Àg2¹- _wandbruntime‚2108  _runtime‚2108ã@Yú'ºôO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184363939904512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚194262300957840 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57855045 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197306518848837 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 _step‚100 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199050756969984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121044992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26520975 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4204524091904 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3045956209152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16785373 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31360408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11466124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 # _timestamp‚1.78847190185248e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202096686787584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16785368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31334070 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚194262275529872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 trainer/global_step‚1249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199050756969984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31360408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11466828 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 _runtime‚2108.609752477 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚202096686787584]ü„þ'»øO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11466124 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 # _timestamp‚1.78847190185248e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202096686787584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16785368 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31334070 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚194262275529872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 trainer/global_step‚1249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199050756969984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31360408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11466828 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 _runtime‚2108.609752477 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚202096686787584 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184363939904512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚194262300957840 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57855045 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197306518848837 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 _step‚100 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199050756969984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121044992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26520975 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4204524091904 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3045956209152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16785373 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31360408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000d0|Ê¿2¼- _wandbruntime‚2111  _runtime‚2111ÖY¿'½ÿM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 118934528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67731456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 trainer/global_step‚1249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚101 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3051977363968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788471904460051e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199204534529024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28273101 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚194412630198976 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57926406 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199204508137472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31385001 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199204534529024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28273776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202252874828288 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57900036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16798173 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197459254393467 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57900036 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57926406 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31358637 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180298706201600 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3046624194491 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184506682905088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26541399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11475598V'¿Û¾ƒN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚194412604771008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199204508137472 _runtime‚2111.222979419 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26541405 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31385001 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199204534529024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocateddðW¿."‚28273776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202252874828288 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57900036 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16798173 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197459254393467 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57900036 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57926406 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31358637 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180298706201600 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3046624194491 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184506682905088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26541399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11475598 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 118934528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67731456 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 trainer/global_step‚1249 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚101 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3051977363968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788471904460051e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199204534529024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28273101 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚194412630198976 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57926406 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199204508137472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31385001 Få¡XÀÒ epoch‚1 .loss/FRt2/detection_loss‚2.347997658760161 /loss/FBt3/detection_loss‚2.4844915591753445 trainer/global_step‚1249 % _timestamp‚1.7884719044701335e+09 1loss/SRqqt3/assignment_loss‚2.418501568833987 0loss/SRqqt2/detection_loss‚2.822600078582764 1loss/SRqqt4/detection_loss‚2.4143400955945253 /loss/FBt4/detection_loss‚2.4372180183728536 0loss/FRt1/assignment_loss‚3.4225397195134843 0loss/FRt2/assignment_loss‚3.0180659600206323 /loss/FRt4/assignment_loss‚5.274949129889993 .loss/FRt3/detection_loss‚3.224390366803045 /loss/FRt3/assignment_loss‚4.776473252669625 2loss/SRqqt1/assignment_loss‚1.6567269624210894 2loss/SRqqt4/assignment_loss‚2.4283730182796717 1loss/FBt2/assignment_loss‚0.24550933344922002 .loss/FRt4/detection_loss‚2.827679900562062 %loss/total_loss‚56.06995245280609 _runtime‚2111.226901263 _step‚102 1loss/FBt3/assignment_loss‚0.45862268494406283 /loss/FRt1/detection_loss‚1.6494941701314279 0loss/SRqqt1/detection_loss‚0.957723411207553 /loss/FBt1/detection_loss‚3.4967269727161954 1loss/FBt4/assignment_loss‚0.30354791730642317 1loss/SRqqt2/assignment_loss‚2.931463845570882 0loss/FBt1/assignment_loss‚0.7108173734681519 1loss/SRqqt3/detection_loss‚2.9344475328922273 .loss/FBt2/detection_loss‚0.827251921640709ŽþD\ÁÖ 0loss/SRqqt2/detection_loss‚2.822600078582764 1loss/SRqqt4/detection_loss‚2.4143400955945253 /loss/FBt4/detection_loss‚2.4372180183728536 0loss/FRt1/assignment_loss‚3.4225397195134843 0loss/FRt2/assignment_loss‚3.0180659600206323 /loss/FRt4/assignment_loss‚5.274949129889993 .loss/FRt3/detection_loss‚3.224390366803045 /loss/FRt3/assignment_loss‚4.776473252669625 2loss/SRqqt1/assignment_loss‚1.6567269624210894 2loss/SRqqt4/assignment_loss‚2.4283730182796717 1loss/FBt2/assignment_loss‚0.24550933344922002 .loss/FRt4/detection_loss‚2.827679900562062 %loss/total_loss‚56.06995245280609 _runtime‚2111.226901263 _step‚102 1loss/FBt3/assignment_loss‚0.45862268494406283 /loss/FRt1/detection_loss‚1.6494941701314279 0loss/SRqqt1/detection_loss‚0.957723411207553 /loss/FBt1/detection_loss‚3.4967269727161954 1loss/FBt4/assignment_loss‚0.30354791730642317 1loss/SRqqt2/assignment_loss‚2.931463845570882 0loss/FBt1/assignment_loss‚0.7108173734681519 1loss/SRqqt3/detection_loss‚2.9344475328922273 .loss/FBt2/detection_loss‚0.827251921640709 epoch‚1 .loss/FRt2/detection_loss‚2.347997658760161 /loss/FBt3/detection_loss‚2.4844915591753445 trainer/global_step‚1249 % _timestamp‚1.7884719044701335e+09 1loss/SRqqt3/assignment_loss‚2.418501568833987fPr¹.œÂ‚(:‘ äÔçÔ±·µz gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.595' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.91900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùY^~sÂ(:è äÔçÔÄ÷Ç{ network.sent‚ 2.152168e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 68.56640625 disk.sda3.in‚0 disk.sda3.out‚ 68.56640625* proc.memory.percent‚20.515823364257812 cpu‚24.70004362372031 network.recv‚686810 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚51787.00439453125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.30859375$ memory_percent‚68.23217868804932" disk.nvme0c0n1.in‚ 4947.1171875# disk.dm-0.out‚51787.00439453125 disk.dm-1.out‚ 68.56640625 disk.sda.in‚0 disk.nvme0n1.in‚ 4947.1171875& disk.nvme0n1.out‚51787.00439453125 proc.memory.rssMB‚ 3361.3125) disk./.usagePercent‚77.70113933388232% disk./.usageGB‚1157.8031272888184 disk.dm-0.in‚ 4947.1171875) proc.memory.availableMB‚ 5204.83984375 proc.cpu.threads‚17ÍrꆒÄ‚(:‡ óÔçÔÓŠ¸w gpu.0.gpu‚46.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚203.021' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.6042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HAWÁmÅ‚(:â óÔçÔؘ¡x% disk./.usageGB‚1157.8030891418457 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# disk.dm-0.out‚52230.53857421875 disk.dm-1.out‚ 68.9921875 disk.sda.in‚0 disk.sda.out‚ 68.9921875" disk.nvme0c0n1.in‚ 4947.1171875 disk.dm-1.in‚0* proc.memory.percent‚20.515823364257812 proc.cpu.threads‚17 network.sent‚ 2.261655e+06 disk.drbd1260.out‚0.3125& disk.nvme0n1.out‚52230.53857421875( disk.nvme0c0n1.out‚52230.53857421875$ memory_percent‚68.23370456695557) proc.memory.availableMB‚ 5204.58984375 proc.memory.rssMB‚ 3361.3125 cpu‚24.703488467668393 network.recv‚694305) disk./.usagePercent‚77.70113677380698 disk.sda3.in‚0 disk.sda3.out‚ 68.9921875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 4947.1171875 disk.sda1.out‚0 disk.dm-0.in‚ 4947.1171875÷Ül”lÆ‚(:á ‚ÕçÔªýöx disk.sda3.out‚ 69.4140625 disk.drbd1260.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 4947.1171875( disk.nvme0c0n1.out‚52505.68701171875# disk.dm-0.out‚52505.68701171875 disk.sda.in‚0 disk.drbd1260.out‚0.3125$ memory_percent‚68.23222637176514) proc.memory.availableMB‚ 5204.83203125 network.recv‚697544) disk./.usagePercent‚77.70113805384466& disk.nvme0n1.out‚52505.68701171875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 4947.1171875 disk.dm-1.in‚0 disk.dm-1.out‚ 69.4140625 disk.sda3.in‚0 proc.memory.rssMB‚ 3361.3125* proc.memory.percent‚20.515823364257812 cpu‚24.706908268693837 network.sent‚ 2.266657e+06$ disk./.usageGB‚1157.803108215332 disk.nvme0n1.in‚ 4947.1171875 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda.out‚ 69.4140625¼ŽÇ‚(:… ‚ÕçÔåêŸy gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚38.0 gpu.0.powerWatts‚146.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.2572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹hDZÈ‚(:Ï ‘ÕçÔöìÍy disk.sda3.out‚ 69.890625 disk.drbd1260.out‚ 0.31640625 disk.sda2.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 3361.3125* proc.memory.percent‚20.515823364257812 network.sent‚ 2.274697e+06 network.recv‚702142% disk./.usageGB‚1157.8030891418457 disk.dm-0.in‚ 4947.1171875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 69.890625 disk.nvme0n1.in‚ 4947.1171875 disk.dm-0.out‚ 52708.03125 disk.drbd1260.in‚0 disk.nvme0n1.out‚ 52708.03125 disk.sda2.out‚0 cpu‚24.71061415638658 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 4947.1171875" disk.nvme0c0n1.out‚ 52708.03125 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚68.23229789733887( proc.memory.availableMB‚ 5204.8203125) disk./.usagePercent‚77.70113677380698 disk.dm-1.out‚ 69.890625§òÒÉ‚(:… ‘ÕçÔïË…{ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…&4aÊ‚(:Ö  ÕçÔÓÑòx network.recv‚705636) disk./.usagePercent‚77.68452956506081 disk.drbd1260.in‚0 disk.nvme0n1.out‚ 53187.09375# disk.nvme0c0n1.in‚ 5059.42578125 disk.sda.out‚ 70.25390625 proc.cpu.threads‚17 network.sent‚ 2.27772e+06% disk./.usageGB‚1157.5556297302246! disk.nvme0n1.in‚ 5059.42578125 disk.sda2.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.out‚ 53187.09375 disk.sda.in‚0' proc.memory.availableMB‚ 5188.703125 disk.drbd1260.out‚ 0.31640625 disk.sda2.out‚0 disk.dm-0.out‚ 53187.09375$ memory_percent‚68.33066940307617 proc.memory.rssMB‚ 3361.3125* proc.memory.percent‚20.515823364257812 disk.sda3.in‚0 disk.sda3.out‚ 70.25390625 disk.sda1.out‚0 disk.dm-0.in‚ 5059.42578125 disk.dm-1.in‚0 disk.dm-1.out‚ 70.25390625 cpu‚24.71386342931481·p +Ë‚(:’  ÕçÔ‰«¢y gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚118.648' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.729599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00(2Ì- _wandbruntime‚2188  _runtime‚2188tEÅÍ¿ % _timestamp‚1.7884719813285155e+09 _runtime‚2188.084182032 _step‚103 % lr-AdamW/pg1‚0.000651742069107883 % lr-AdamW/pg2‚0.000651742069107883 trainer/global_step‚1299.)0ÉÎà _step‚103 % lr-AdamW/pg1‚0.000651742069107883 % lr-AdamW/pg2‚0.000651742069107883 trainer/global_step‚1299 % _timestamp‚1.7884719813285155e+09 _runtime‚2188.084182032gDvb2Ï- _wandbruntime‚2188  _runtime‚2188¶Ž+Üü'ÐöO KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3166494706176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60134291 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3170105366016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27542175 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187123986465280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4371330546688 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201778763014640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60107947 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206739608530432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17425618 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123142144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27542181 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2188.085529803 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17425623 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71939072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11911210 % _timestamp‚1.7884719813296988e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204939675570501 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206739634921984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32565772 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11910491 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206739634921984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4371402485760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201778737586672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209909740288000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27542175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209906103236608 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191495265808896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3166494706176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29336109 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209906103236608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32592110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚724 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191495388951040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚254279689*G¡€ÑúO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27542175 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187123986465280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4371330546688 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201778763014640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60107947 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206739608530432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17425618 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123142144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27542181 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2188.085529803 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17425623 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71939072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11911210 % _timestamp‚1.7884719813296988e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204939675570501 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206739634921984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32565772 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11910491 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206739634921984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4371402485760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201778737586672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209909740288000 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27542175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209906103236608 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191495265808896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3166494706176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29336109 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209906103236608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32592110 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚724 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191495388951040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3166494706176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60134291 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3170105366016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877hkU—–2Ò- _wandbruntime‚2189  _runtime‚2189­Aà'Ó€N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27562599 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191637903226368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚210062291277312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27562605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4374747403776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205096045205775 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60179308 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206893386089472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27562599 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32590339 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17438423 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3172515860992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32616703 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32616703 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206893412481024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚210065928342016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11919406 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60152938 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17438428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69828608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187263276854272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205092411114923 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60152938 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29358511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60179308 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3166952949999 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3168905187840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29357829 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3168905187840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27562605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32590339 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121031680 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206893386089472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚201929092255776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1299ÈáD¨ 'Ô„N trainer/global_step‚1299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27562599 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191637903226368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚210062291277312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27562605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4374747403776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚205096045205775 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60179308 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206893386089472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27562599 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32590339 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17438423 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3172515860992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32616703 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32616703 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206893412481024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚210065928342016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11919406 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60152938 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17438428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69828608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187263276854272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚205092411114923 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60152938 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29358511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60179308 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3166952949999 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3168905187840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29357829 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3168905187840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27562605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32590339 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121031680 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206893386089472 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚201929092255776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0i”ýŒ2Õ- _wandbruntime‚2189  _runtime‚2189Yyú£TÖÎ 1loss/FBt2/assignment_loss‚0.30681044353987075 0loss/FBt3/assignment_loss‚0.5566007779347465 /loss/FRt3/assignment_loss‚4.640149839719137 0loss/FBt1/assignment_loss‚1.1487891525030136 0loss/FBt4/assignment_loss‚0.5274058850972276 /loss/FRt2/detection_loss‚2.4069002205505967 /loss/FBt2/detection_loss‚0.8781697699567304 .loss/FBt3/detection_loss‚2.698009945844349 2loss/SRqqt3/assignment_loss‚2.1024925612351475 /loss/FRt3/detection_loss‚2.9826214691003163 epoch‚1 2loss/SRqqt1/assignment_loss‚1.5557455408697327 2loss/SRqqt4/assignment_loss‚2.0093546266503193 /loss/FRt4/detection_loss‚2.6862771827727556 .loss/FBt4/detection_loss‚2.638573629291434 % _timestamp‚1.7884719823437123e+09 .loss/FRt2/assignment_loss‚3.74630464737614 0loss/SRqqt4/detection_loss‚2.506632569081643 _runtime‚2189.099164522 trainer/global_step‚1299 0loss/SRqqt2/assignment_loss‚2.61324496780123 /loss/FRt1/detection_loss‚1.9307232802887613 1loss/SRqqt2/detection_loss‚3.3046276654515947 %loss/total_loss‚57.64388905617463 /loss/FRt1/assignment_loss‚2.957164490222931 0loss/SRqqt3/detection_loss‚2.625156980227022 .loss/FBt1/detection_loss‚4.655529415607452 _step‚106 1loss/SRqqt1/detection_loss‚1.0682269386326273 /loss/FRt4/assignment_loss‚5.098377056419849l¦X×Ò 2loss/SRqqt1/assignment_loss‚1.5557455408697327 2loss/SRqqt4/assignment_loss‚2.0093546266503193 /loss/FRt4/detection_loss‚2.6862771827727556 .loss/FBt4/detection_loss‚2.638573629291434 % _timestamp‚1.7884719823437123e+09 .loss/FRt2/assignment_loss‚3.74630464737614 0loss/SRqqt4/detection_loss‚2.506632569081643 _runtime‚2189.099164522 trainer/global_step‚1299 0loss/SRqqt2/assignment_loss‚2.61324496780123 /loss/FRt1/detection_loss‚1.9307232802887613 1loss/SRqqt2/detection_loss‚3.3046276654515947 %loss/total_loss‚57.64388905617463 /loss/FRt1/assignment_loss‚2.957164490222931 0loss/SRqqt3/detection_loss‚2.625156980227022 .loss/FBt1/detection_loss‚4.655529415607452 _step‚106 1loss/SRqqt1/detection_loss‚1.0682269386326273 /loss/FRt4/assignment_loss‚5.098377056419849 1loss/FBt2/assignment_loss‚0.30681044353987075 0loss/FBt3/assignment_loss‚0.5566007779347465 /loss/FRt3/assignment_loss‚4.640149839719137 0loss/FBt1/assignment_loss‚1.1487891525030136 0loss/FBt4/assignment_loss‚0.5274058850972276 /loss/FRt2/detection_loss‚2.4069002205505967 /loss/FBt2/detection_loss‚0.8781697699567304 .loss/FBt3/detection_loss‚2.698009945844349 2loss/SRqqt3/assignment_loss‚2.1024925612351475 /loss/FRt3/detection_loss‚2.9826214691003163 epoch‚1jÓæïxœØ‚(:‘ ¯ÕçÔ–þÛy gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚47.0 gpu.0.powerWatts‚277.953' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.59059999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jh†ÂqÙ‚(:æ ¯ÕçÔ´Õ‘z disk.dm-0.in‚ 5155.42578125# disk.dm-0.out‚53819.50634765625 disk.dm-1.out‚ 70.77734375 disk.sda.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚53819.50634765625 disk.sda2.in‚0 disk.sda1.out‚0 network.recv‚710640) disk./.usagePercent‚77.55666352999408! disk.nvme0n1.in‚ 5155.42578125$ memory_percent‚69.48299407958984 proc.memory.rssMB‚ 3361.3125* proc.memory.percent‚20.515823364257812 cpu‚24.71647818970509 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚53819.50634765625 proc.cpu.threads‚17 network.sent‚ 2.284056e+06$ disk./.usageGB‚1155.650333404541 disk.sda.out‚ 70.77734375 disk.sda3.out‚ 70.77734375 disk.drbd1260.out‚ 0.3203125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5155.42578125& proc.memory.availableMB‚ 4999.90625ç¹}’Ú‚(:‡ ¾ÕçÔä¤Òv gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚205.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.0754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê­E!vÛ‚(:ë ¾ÕçÔã…¶w proc.cpu.threads‚17) disk./.usagePercent‚77.55508933966605 disk.drbd1260.out‚ 0.3203125! disk.nvme0n1.in‚ 5251.42578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 5251.42578125* proc.memory.percent‚20.515823364257812 network.sent‚ 2.392104e+06& disk.nvme0n1.out‚54107.67529296875( disk.nvme0c0n1.out‚54107.67529296875$ memory_percent‚70.62828540802002) proc.memory.availableMB‚ 4812.26171875 proc.memory.rssMB‚ 3361.3125 network.recv‚716984% disk./.usageGB‚1155.6268768310547 disk.dm-1.in‚0 disk.sda.out‚ 71.20703125 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5251.42578125 disk.dm-1.out‚ 71.20703125 disk.sda.in‚0 disk.sda3.out‚ 71.20703125 disk.drbd1260.in‚0# disk.dm-0.out‚54107.67529296875 cpu‚24.718821299752364÷:’Ü‚(:‡ ÍÕçÔº’‹w gpu.0.gpu‚89.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚47.0 gpu.0.powerWatts‚273.001' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ1ai^Ý‚(:å ÍÕçÔ䌪x& disk.nvme0n1.out‚54203.82275390625 disk.sda2.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 5355.42578125 disk.sda.out‚ 71.5703125 disk.sda3.out‚ 71.5703125 disk.drbd1260.out‚ 0.32421875 proc.memory.rssMB‚ 3361.3125 network.recv‚721598 disk.sda2.out‚0 disk.dm-0.in‚ 5355.42578125 disk.sda3.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚20.515823364257812 cpu‚24.721053882325002( disk.nvme0c0n1.out‚54203.82275390625 network.sent‚ 2.400135e+06( disk./.usagePercent‚77.5566696741749! disk.nvme0n1.in‚ 5355.425«™à78125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 71.5703125& proc.memory.availableMB‚ 4599.59375% disk./.usageGB‚1155.6504249572754# disk.dm-0.out‚54203.82275390625 disk.sda.in‚0$ memory_percent‚71.92630767822266 proc.cpu.threads‚17ÒDIfÞ‚(:Û ÜÕçÔóÕx disk.sda.out‚ 72.0234375 disk.sda3.out‚ 72.0234375 network.sent‚ 2.4032e+06 network.recv‚725001( disk.nvme0c0n1.out‚54323.66455078125# disk.dm-0.out‚54323.66455078125 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 72.0234375 disk.sda.in‚0& disk.nvme0n1.out‚54323.66455078125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚73.19631576538086 proc.memory.rssMB‚ 3361.3125) disk./.usagePercent‚77.55650198923998 disk.nvme0c0n1.in‚ 5591.03125 disk.dm-0.in‚ 5591.03125 disk.drbd1260.out‚ 0.32421875' proc.memory.availableMB‚ 4391.515625* proc.memory.percent‚20.515823364257812 cpu‚24.72413815786479 proc.cpu.threads‚17% disk./.usageGB‚1155.6479263305664 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 5591.03125 disk.sda2.in‚0µ’@u’ß‚(:‡ ÜÕçÔ ‹z gpu.0.gpu‚55.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚213.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.7396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]«÷à‚(:’ ëÕçÔ¨»Þw gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚46.0 gpu.0.powerWatts‚277.102' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.420399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L™Èiá‚(:Þ ëÕçÔÖŸƒz disk.nvme0c0n1.in‚ 5695.03125* proc.memory.percent‚20.515823364257812 proc.cpu.threads‚17# disk.dm-0.out‚54493.89892578125 disk.sda3.out‚ 72.5703125 disk.drbd1260.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 3361.3125 disk.sda3.in‚0 network.sent‚ 2.409579e+06 network.recv‚729812) disk./.usagePercent‚77.54971190140932% disk./.usageGB‚1155.5467491149902 disk.dm-1.out‚ 72.5703125& disk.nvme0n1.out‚54493.89892578125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.328125 disk.sda2.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚54493.89892578125$ memory_percent‚74.46587085723877) proc.memory.availableMB‚ 4183.51171875 cpu‚24.727035005844986 disk.dm-0.in‚ 5695.03125 disk.sda.in‚0 disk.sda.out‚ 72.5703125 disk.nvme0n1.in‚ 5695.03125Z–Çg2â- _wandbruntime‚2252  _runtime‚2252–àþÇãÁ _step‚107 & lr-AdamW/pg1‚0.0006515858747038339 & lr-AdamW/pg2‚0.0006515858747038339 trainer/global_step‚1349 % _timestamp‚1.7884720455099645e+09 _runtime‚2252.266511447ü¹ºŠËäÅ _runtime‚2252.266511447 _step‚107 & lr-AdamW/pg1‚0.0006515858747038339 & lr-AdamW/pg2‚0.0006515858747038339 trainer/global_step‚1349 % _timestamp‚1.7884720455099645e+09k¡W­}2å- _wandbruntime‚2252  _runtime‚2252òd-ú'æôO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30420864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚217719156737024 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12354996 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33797474 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626468737536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3281266739337 _step‚108 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3287033203200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28563375 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123142144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4537962822656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28563381 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚716 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33797474 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198626345595392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194088505914880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212576466382809 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209295225071440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4537890883584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209295199643472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217715519685632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194088454711808 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217715519685632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3287033203200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18065868 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62387193 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212580100473453 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214428486482432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3290643863040 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33797474 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198626345595392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194088505914880 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212576466382809 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚209295225071440 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4537890883584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209295199643472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217715519685632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194088454711808 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217715519685632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1349 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3287033203200 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18065868 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62387193 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212580100473453 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214428486482432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3290643863040 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33797474 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626468737536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3281266739337 _step‚108 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚721 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3287033203200 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28563375 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123142144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4537962822656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28563381 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚716 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144ló¨2è- _wandbruntime‚2253  _runtime‚2253n­ÌØu"é€N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1349 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30442479 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18078673 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198769094525952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3287281705507 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62432210 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30443191 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62432210 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28583799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚214582290433024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28583799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3289443684864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194227745100800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33848405 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚214582264041472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62405840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 _runtime‚2253.286460977 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198768971397120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217875344791040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚214582290433024 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33822041 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4541298222080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209445528884608 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3289443684864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209445554312576 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28583805 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18078678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12363806 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212729201927231 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33822041 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212732836018083 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62405840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4541226296320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3283673042623 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12364513 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217875344791040 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 % _timestamp‚1.7884720465303879e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33848405 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚707 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62405840 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4541226296320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3283673042623 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12364513 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217875344791040 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 % _timestamp‚1.7884720465303879e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33848405 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30442479 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18078673 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198769094525952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3287281705507 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62432210 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30443191 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62432210 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28583799 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚214582290433024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28583799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3289443684864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194227745100800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33848405 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚214582264041472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62405840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 _runtime‚2253.286460977 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198768971397120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217875344791040 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚214582290433024 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33822041 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912m‡ß[2ë- _wandbruntime‚2253  _runtime‚2253m²òoRìÌ 0loss/FRt1/assignment_loss‚2.9328039677246758 /loss/FRt2/detection_loss‚2.6260139124863073 0loss/SRqqt2/detection_loss‚3.310487115383148 % _timestamp‚1.7884720465326066e+09 /loss/FRt2/assignment_loss‚3.732518510384993 1loss/FBt1/assignment_loss‚0.36758362371474507 0loss/FBt4/assignment_loss‚0.7333142990867297 .loss/FRt3/detection_loss‚3.292876620041697 0loss/SRqqt1/detection_loss‚1.165824459466551 /loss/FBt4/detection_loss‚2.4854774594306948 trainer/global_step‚1349 0loss/FBt3/assignment_loss‚0.6872198065698502 .loss/FRt4/detection_loss‚3.095288286286016 _runtime‚2253.287941879 0loss/SRqqt4/detection_loss‚2.731166443824768 /loss/FBt2/detection_loss‚1.1566681413080644 .loss/FBt3/detection_loss‚2.308465531894139 epoch‚1 1loss/SRqqt2/assignment_loss‚2.703488826751709 /loss/FRt1/detection_loss‚1.8263495333789699 /loss/FRt3/assignment_loss‚4.732948127545808 %loss/total_loss‚59.36381415037586 _step‚110 2loss/SRqqt3/assignment_loss‚2.3997806033917835 0loss/SRqqt4/assignment_loss‚2.10313782453537 0loss/FBt2/assignment_loss‚0.3128671719791813 .loss/FRt4/assignment_loss‚4.58455785628288 1loss/SRqqt1/assignment_loss‚1.466693854524887 1loss/SRqqt3/detection_loss‚2.6877652066094533 /loss/FBt1/detection_loss‚5.9205169677734375üÍVíÐ 0loss/FRt1/assignment_loss‚2.9328039677246758 /loss/FRt2/detection_loss‚2.6260139124863073 0loss/SRqqt2/detection_loss‚3.310487115383148 % _timestamp‚1.7884720465326066e+09 /loss/FRt2/assignment_loss‚3.732518510384993 1loss/FBt1/assignment_loss‚0.36758362371474507 0loss/FBt4/assignment_loss‚0.7333142990867297 .loss/FRt3/detection_loss‚3.292876620041697 0loss/SRqqt1/detection_loss‚1.165824459466551 /loss/FBt4/detection_loss‚2.4854774594306948 trainer/global_step‚1349 0loss/FBt3/assignment_loss‚0.6872198065698502 .loss/FRt4/detection_loss‚3.095288286286016 _runtime‚2253.287941879 0loss/SRqqt4/detection_loss‚2.731166443824768 /loss/FBt2/detection_loss‚1.1566681413080644 .loss/FBt3/detection_loss‚2.308465531894139 epoch‚1 1loss/SRqqt2/assignment_loss‚2.703488826751709 /loss/FRt1/detection_loss‚1.8263495333789699 /loss/FRt3/assignment_loss‚4.732948127545808 %loss/total_loss‚59.36381415037586 _step‚110 2loss/SRqqt3/assignment_loss‚2.3997806033917835 0loss/SRqqt4/assignment_loss‚2.10313782453537 0loss/FBt2/assignment_loss‚0.3128671719791813 .loss/FRt4/assignment_loss‚4.58455785628288 1loss/SRqqt1/assignment_loss‚1.466693854524887 1loss/SRqqt3/detection_loss‚2.6877652066094533 /loss/FBt1/detection_loss‚5.9205169677734375nÐÅj“›î‚(: úÕçÔ«y gpu.0.gpu‚8.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚45.0 gpu.0.powerWatts‚266.962' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.392399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^zy¼iï‚(:Þ úÕçÔ…ôœy disk.dm-1.in‚0 disk.sda3.in‚0# disk.dm-0.out‚54972.94189453125 disk.dm-1.out‚ 72.93359375& proc.memory.availableMB‚ 3963.21875 proc.memory.rssMB‚ 3361.3125 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 72.93359375$ memory_percent‚75.81043243408203* proc.memory.percent‚20.515823364257812 cpu‚24.729316572510506 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 2.414433e+06) disk./.usagePercent‚77.56064444715153 disk.sda3.out‚ 72.93359375 disk.nvme0n1.in‚ 5807.03125& disk.nvme0n1.out‚54972.94189453125 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚733070% disk./.usageGB‚1155.7096519470215 disk.drbd1260.in‚0 disk.drbd1260.out‚0.328125 disk.nvme0c0n1.in‚ 5807.03125( disk.nvme0c0n1.out‚54972.94189453125 disk.dm-0.in‚ 5807.03125ˆPmð‚(:â ‰ÖçÔí¢·x( disk./.usagePercent‚77.5606203824433# disk.dm-0.out‚55380.31689453125 disk.dm-1.out‚ 73.36328125 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0* proc.memory.percent‚20.515823364257812 network.recv‚740235 disk.dm-0.in‚ 5911.03125& disk.nvme0n1.out‚55380.31689453125 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚77.10611820220947 proc.cpu.threads‚17% disk./.usageGB‚1155.7092933654785 disk.sda3.in‚0 disk.drbd1260.out‚ 0.33203125) proc.memory.availableMB‚ 3750.93359375 cpu‚24.731778381463283 disk.nvme0c0n1.in‚ 5911.03125( disk.nvme0c0n1.out‚55380.31689453125 disk.dm-1.in‚0 disk.sda.out‚ 73.36328125 disk.sda3.out‚ 73.36328125 disk.nvme0n1.in‚ 5911.03125 disk.sda2.in‚0 proc.memory.rssMB‚ 3361.3125 network.sent‚ 2.524009e+06’w¯œñ‚(:‘ ‰ÖçÔ‚Žèx gpu.0.gpu‚12.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)X`’ò‚(:‡ ˜ÖçÔÿÏáw gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚46.0 gpu.0.powerWatts‚265.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 “*ïpó‚(:å ˜ÖçÔ¾ð±x disk.sda.in‚0( proc.memory.availableMB‚ 3542.9453125 proc.cpu.threads‚17 network.recv‚742652 disk.sda1.out‚0 disk.dm-0.in‚ 6150.64453125 disk.sda.out‚ 73.796875 disk.drbd1260.out‚ 0.33203125$ memory_percent‚78.37557792663574! disk.nvme0n1.in‚ 6150.64453125& disk.nvme0n1.out‚55667.92626953125 disk.sda2.in‚0# disk.dm-0.out‚55667.92626953125 disk.dm-1.out‚ 73.796875 disk.sda3.in‚0 proc.memory.rssMB‚ 3361.3125* proc.memory.percent‚20.515823364257812) disk./.usagePercent‚77.56094192790631% disk./.usageGB‚1155.7140846252441 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 73.796875 disk.drbd1260.in‚0 cpu‚24.734089108183767 network.sent‚ 2.527853e+06# disk.nvme0c0n1.in‚ 6150.64453125( disk.nvme0c0n1.out‚55667.926269531253ÕKq’ô‚(:‡ §ÖçÔÀ¦ðx gpu.0.gpu‚71.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚135.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.1718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—-¯quõ‚(:ê §ÖçÔ»¸Œz) disk./.usagePercent‚77.56093834380083 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 6254.64453125 disk.dm-0.in‚ 6254.64453125( proc.memory.availableMB‚ 3338.7109375 proc.memory.rssMB‚ 3361.3125 network.sent‚ 2.534238e+06% disk./.usageGB‚1155.7140312194824 disk.sda3.out‚ 74.45703125! disk.nvme0n1.in‚ 6254.64453125 disk.sda2.out‚0( disk.nvme0c0n1.out‚55673.33837890625 disk.dm-1.in‚0 disk.dm-1.out‚ 74.45703125 disk.sda.in‚0 disk.sda.out‚ 74.45703125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3359375& disk.nvme0n1.out‚55673.33837890625# disk.dm-0.out‚55673.33837890625$ memory_percent‚79.62212562561035 network.recv‚747866 disk.sda1.out‚0* proc.memory.percent‚20.515823364257812 cpu‚24.736126276713833 proc.cpu.threads‚17 Ú‡ò2ö- _wandbruntime‚2324  _runtime‚2324Q°-‚Ç÷Á & lr-AdamW/pg1‚0.0006513618132803518 & lr-AdamW/pg2‚0.0006513618132803518 trainer/global_step‚1399 % _timestamp‚1.7884721181075726e+09 _runtime‚2324.863196823 _step‚111ÑäÐ?ËøÅ & lr-AdamW/pg2‚0.0006513618132803518 trainer/global_step‚1399 % _timestamp‚1.7884721181075726e+09 _runtime‚2324.863196823 _step‚111 & lr-AdamW/pg1‚0.0006513618132803518oÅv I2ù- _wandbruntime‚2324  _runtime‚2324 + ‘ü'úöO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31507323 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205757398462976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 trainer/global_step‚1399 _runtime‚2324.864327154 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64613751 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220213257195117 _step‚112 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225524936134656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚222117364434432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216811687128240 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35029176 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚225528573186048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3411182360064 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35029176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225524936134656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123142144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3405204157521 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216811661700272 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3407571700224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3401595494845 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29584581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12801205 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18706118 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4704301159424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3411182360064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35055514 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201052974161408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71939072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220216891285761 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4704373098496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205757275320832 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35055514 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31508034 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201053025364480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225528573186048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3407571700224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29584581 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29584575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222117390825984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222117364434432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18706123 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222117390825984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 % _timestamp‚1.7884721181084316e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12801911 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225524936134656 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚222117364434432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216811687128240 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35029176 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚225528573186048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3411182360064 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35029176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚225524936134656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 VHDeviceStatsMonitor.on_‘ÿ8Ltrain_batch_start/inactive_split_bytes.all.current‚ 123142144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3405204157521 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216811661700272 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3407571700224 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3401595494845 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29584581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12801205 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18706118 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4704301159424 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3411182360064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35055514 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201052974161408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71939072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220216891285761 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4704373098496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205757275320832 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35055514 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31508034 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201053025364480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225528573186048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3407571700224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29584581 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29584575 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222117390825984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222117364434432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18706123 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222117390825984 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 % _timestamp‚1.7884721181084316e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12801911 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31529080 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚222271168385024 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚222271141993472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3404001798131 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3409982181888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 trainer/global_step‚1399 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64658742 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4707711382528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201192315753472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3413592855040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12810865 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201192264550400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216961990941408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222271141993472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29605005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29605005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205900027136000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225684761240064 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64685112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35053743 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220369626830391 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚220365992739539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3407610461015 % _timestamp‚1.7884721203765433e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35080107 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚113 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35053743 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205899904007168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4707639456768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29604999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18718923 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12810157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64685112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31529793 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925760 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18718928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64685112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35053743 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220369626830391 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚220365992739539 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3407610461015 % _timestamp‚1.7884721203765433e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35080107 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚113 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35053743 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205899904007168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4707639456768 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29604999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18718923 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12810157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64685112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31529793 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925760 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18718928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31529080 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚222271168385024 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚222271141993472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 499936768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3404001798131 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3409982181888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 trainer/global_step‚1399 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64658742 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4707711382528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201192315753472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3413592855040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12810865 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201192264550400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216961990941408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222271141993472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29605005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29605005 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205900027136000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225684761240064q†ò!ž2- _wandbruntime‚2327  _runtime‚2327ˆÚÆV‚Ð 2loss/SRqqt1/assignment_loss‚1.4545267384985219 1loss/SRqqt4/assignment_loss‚2.304454100228125 1loss/FBt3/assignment_loss‚0.46578475447216383 0loss/FBt4/assignment_loss‚0.4717892893432994 1loss/SRqqt3/detection_loss‚2.0660368603738872 /loss/FBt2/detection_loss‚0.7237621329673757 .loss/FBt3/detection_loss‚2.449861450990041 # _timestamp‚1.78847212038366e+09 0loss/FRt1/assignment_loss‚3.0167755055076935 /loss/FRt4/assignment_loss‚4.629753910005093 1loss/SRqqt3/assignment_loss‚2.115622240711342 1loss/SRqqt2/detection_loss‚2.5988726139068605 .loss/FBt1/detection_loss‚5.817767688206264 0loss/FRt2/assignment_loss‚3.3120614759465483 1loss/FBt2/assignment_loss‚0.23764035926513016 /loss/FRt1/detection_loss‚1.7101820751905668 %loss/total_loss‚56.41160749835747 epoch‚1 trainer/global_step‚1399 /loss/FRt3/assignment_loss‚5.635408977667491 /loss/FRt2/detection_loss‚2.3205284628462284 .loss/FRt3/detection_loss‚2.953717521826426 0loss/FBt1/assignment_loss‚0.3291084942008768 .loss/FRt4/detection_loss‚2.766239103861153 0loss/SRqqt4/detection_loss‚2.517919542328004 2loss/SRqqt2/assignment_loss‚2.8239977419376374 _step‚114 1loss/SRqqt1/detection_loss‚1.0994371716095053 _runtime‚2327.139753536 /loss/FBt4/detection_loss‚2.5903592864672342aO•ïZƒÔ /loss/FBt4/detection_loss‚2.5903592864672342 2loss/SRqqt1/assignment_loss‚1.4545267384985219 1loss/SRqqt4/assignment_loss‚2.304454100228125 1loss/FBt3/assignment_loss‚0.46578475447216383 0loss/FBt4/assignment_loss‚0.4717892893432994 1loss/SRqqt3/detection_loss‚2.0660368603738872 /loss/FBt2/detection_loss‚0.7237621329673757 .loss/FBt3/detection_loss‚2.449861450990041 # _timestamp‚1.78847212038366e+09 0loss/FRt1/assignment_loss‚3.0167755055076935 /loss/FRt4/assignment_loss‚4.629753910005093 1loss/SRqqt3/assignment_loss‚2.115622240711342 1loss/SRqqt2/detection_loss‚2.5988726139068605 .loss/FBt1/detection_loss‚5.817767688206264 0loss/FRt2/assignment_loss‚3.3120614759465483 1loss/FBt2/assignment_loss‚0.23764035926513016 /loss/FRt1/detection_loss‚1.7101820751905668 %loss/total_loss‚56.41160749835747 epoch‚1 trainer/global_step‚1399 /loss/FRt3/assignment_loss‚5.635408977667491 /loss/FRt2/detection_loss‚2.3205284628462284 .loss/FRt3/detection_loss‚2.953717521826426 0loss/FBt1/assignment_loss‚0.3291084942008768 .loss/FRt4/detection_loss‚2.766239103861153 0loss/SRqqt4/detection_loss‚2.517919542328004 2loss/SRqqt2/assignment_loss‚2.8239977419376374 _step‚114 1loss/SRqqt1/detection_loss‚1.0994371716095053 _runtime‚2327.139753536r¿‰-â„‚(:’ ÅÖçÔÒ«îw gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.372600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ˜Á÷â…‚(:Þ ÅÖçÔ³ç†x disk.sda1.out‚0 disk.dm-0.out‚56319.73828125 disk.sda.in‚0 disk.sda3.out‚ 75.2734375 disk.drbd1260.out‚ 0.33984375 network.sent‚ 2.637015e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 75.2734375 disk.drbd1260.in‚0$ memory_percent‚82.06629753112793* proc.memory.percent‚20.515823364257812 cpu‚24.740137364019198 proc.cpu.threads‚17 network.recv‚757958 disk.sda2.out‚0% disk.nvme0c0n1.out‚56319.73828125 disk.sda.out‚ 75.2734375) disk./.usagePercent‚77.55767731982968! disk.nvme0n1.in‚ 6454.64453125 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6454.64453125 disk.dm-0.in‚ 6454.64453125 disk.sda3.in‚0( proc.memory.availableMB‚ 2938.íf‡2578125 proc.memory.rssMB‚ 3361.3125$ disk./.usageGB‚1155.665439605713# disk.nvme0n1.out‚56319.73828125 disk.sda1.in‚0‹õEb†‚(:× ÔÖçÔÁì­x# disk.nvme0c0n1.out‚ 56641.890625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.33984375 disk.sda2.out‚0) disk./.usagePercent‚77.57029644320745 disk.sda1.out‚0 disk.dm-0.in‚ 6632.29296875 disk.sda.out‚ 75.67578125! disk.nvme0n1.in‚ 6632.29296875 disk.sda2.in‚0# memory_percent‚83.3627700805664 proc.memory.rssMB‚ 3361.3125 network.sent‚ 2.734376e+06 disk.sda1.in‚0 disk.dm-0.out‚ 56641.890625! disk.nvme0n1.out‚ 56641.890625* proc.memory.percent‚20.515823364257812 cpu‚24.742027099221357 proc.cpu.threads‚17 network.recv‚763506 disk.dm-1.out‚ 75.67578125 disk.sda.in‚0 disk.sda3.out‚ 75.67578125 disk.drbd1260.in‚0& proc.memory.availableMB‚ 2725.84375# disk./.usageGB‚1155.85347366333# disk.nvme0c0n1.in‚ 6632.29296875Ë{Ÿ‡‚(:… ÔÖçÔŠßx gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¯Ú{‘ˆ‚(:† ãÖçÔ‚©Åx gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚179.695' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.939 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞGÓ«u‰‚(:ê ãÖçÔçýËy disk.drbd1260.out‚0.34375 disk.sda2.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 6829.39453125( disk.nvme0c0n1.out‚56951.76708984375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 76.24609375) disk./.usagePercent‚77.57026213819786 disk.dm-0.in‚ 6829.39453125# disk.dm-0.out‚56951.76708984375( proc.memory.availableMB‚ 2505.3828125" proc.memory.rssMB‚ 3361.3203125 cpu‚24.743874799602484 network.sent‚ 2.740801e+06! disk.nvme0n1.in‚ 6829.39453125& disk.nvme0n1.out‚56951.76708984375 disk.sda1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚768376% disk./.usageGB‚1155.8529624938965 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 76.24609375 disk.sda.out‚ 76.24609375# memory_percent‚84.7083568572998* proc.memory.percent‚20.515871047973633LjÚÔuŠ‚(:ê òÖçÔ™•µy disk.sda.out‚ 76.67578125 network.sent‚ 2.74561e+06) disk./.usagePercent‚77.58704266404472% disk./.usageGB‚1156.1030044555664# disk.nvme0c0n1.in‚ 6925.39453125 disk.drbd1260.out‚0.34375& disk.nvme0n1.out‚57273.89208984375 disk.sda2.out‚0# disk.dm-0.out‚57273.89208984375 disk.sda3.in‚0 disk.sda3.out‚ 76.67578125 disk.sda1.in‚0$ memory_percent‚85.90474128723145* proc.memory.percent‚20.515871047973633 cpu‚24.745829864470387 network.recv‚771926( disk.nvme0c0n1.out‚57273.89208984375 disk.dm-0.in‚ 6925.39453125 disk.dm-1.out‚ 76.67578125 disk.drbd1260.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 6925.39453125 disk.sda2.in‚0( proc.memory.availableMB‚ 2309.3671875" proc.memory.rssMB‚ 3361.3203125c¼Jð’‹‚(:‡ òÖçÔ×éÄy gpu.0.gpu‚53.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚197.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.4346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆöW’Œ‚(:‡ ×çÔ«õõw gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚149.618' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”&8+v‚(:ë ×çÔ¼ÛÚx! disk.nvme0n1.in‚ 7029.39453125 disk.sda2.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚57486.55224609375 disk.dm-1.out‚ 77.18359375# memory_percent‚87.1511459350586* proc.memory.percent‚20.515871047973633% disk./.usageGB‚1156.1029472351074 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.34765625# disk.nvme0c0n1.in‚ 7029.39453125& proc.memory.availableMB‚ 2105.15625 proc.cpu.threads‚17 network.sent‚ 2.751764e+06 disk.sda3.in‚0& disk.nvme0n1.out‚57486.55224609375 disk.dm-0.in‚ 7029.39453125" proc.memory.rssMB‚ 3361.3203125 cpu‚24.747672156511697 network.recv‚776677 disk.sda.in‚0 disk.sda.out‚ 77.18359375 disk.sda3.out‚ 77.18359375 disk.sda2.out‚0 disk.sda1.out‚0# disk.dm-0.out‚57486.55224609375 disk.dm-1.in‚0) disk./.usagePercent‚77.58703882393169) h¥2Ž- _wandbruntime‚2404  _runtime‚2404×¹îÇÁ & lr-AdamW/pg1‚0.0006510699315315016 & lr-AdamW/pg2‚0.0006510699315315016 trainer/global_step‚1449 % _timestamp‚1.7884721974164238e+09 _runtime‚2404.173390486 _step‚1158‰µ ËÅ & lr-AdamW/pg1‚0.0006510699315315016 & lr-AdamW/pg2‚0.0006510699315315016 trainer/global_step‚1449 % _timestamp‚1.7884721974164238e+09 _runtime‚2404.173390486 _step‚115s±yZ†2‘- _wandbruntime‚2404  _runtime‚2404 ³r‘ü'’öO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3528110197248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚224328149185040 _step‚116 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36260878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 trainer/global_step‚1449 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13245186 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3531720857088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208017544814080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19346373 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233334352583680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚233334352583680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30605775 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30605775 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32591559 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4870603735552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3531720857088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212888220488704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36260878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212888097346560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233337989635072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3528110197248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13244484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123142144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229806242386432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3525532913029 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30605781 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36287216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19346368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4870675674624 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 % _timestamp‚1.7884721974198554e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229806268777984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71939072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 % _timestamp‚1.7884721974198554e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229806268777984 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71939072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 trainer/global_step‚1449 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13245186 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3531720857088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208017544814080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19346373 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233334352583680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚233334352583680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 499936768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30605775 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30605775 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32591559 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4870603735552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3531720857088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212888220488704 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36260878 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212888097346560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233337989635072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3528110197248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13244484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123142144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229806242386432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3525532913029 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30605781 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36287216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19346368t÷1}2”- _wandbruntime‚2406  _runtime‚2406:N³àו€N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30626199 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4873980794368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3524330553639 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1449 _runtime‚2406.673028778 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32612512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233494177689088 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224478478426176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30626199 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233490540624384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208156835203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228002783551847 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229960046337024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128832 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66938014 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36311809 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4873908868608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66911644 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208156784000000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66911644 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229960046337024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229960019945472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13253339 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213030692868608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66938014 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30626205 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36285445 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3530520678912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32613220 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185y|Î0 '–„N XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3530520678912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128832 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66938014 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36311809 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4873908868608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66911644 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208156784000000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66911644 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229960046337024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229960019945472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13253339 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213030692868608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66938014 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30626205 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36285445 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3530520678912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32613220 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30626199 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4873980794368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3524330553639 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1449 _runtime‚2406.673028778 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32612512 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233494177689088 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224478478426176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30626199 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33390854144 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233490540624384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208156835203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228002783551847 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229960046337024 F‚(:’ Ÿ×çÔËû¯x gpu.0.gpu‚88.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚145.282' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.056400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@'´t’ž‚(:‡ ®×çÔýýìx gpu.0.gpu‚55.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚41.0 gpu.0.powerWatts‚204.897' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.9794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÄyomŸ‚(:â ®×çÔ×æ‹y disk.sda.out‚ 78.68359375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3515625 disk.sda1.out‚0& disk.nvme0c0n1.out‚58473.603515625 network.sent‚ 2.871182e+06% disk./.usageGB‚1156.6029510498047 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 7422.6640625! disk.dm-0.out‚58472.353515625$ memory_percent‚90.62480926513672 disk.nvme0n1.in‚ 7422.6640625) disk./.usagePercent‚77.62059449946717 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 7422.6640625& proc.memory.availableMB‚ 1536.03125* proc.memory.percent‚20.515871047973633 proc.cpu.threads‚17 disk.dm-1.out‚ 78.68359375 disk.sda3.out‚ 78.68359375$ disk.nvme0n1.out‚58474.103515625" proc.memory.rssMB‚ 3361.3203125 cpu‚24.753218322668772 network.recv‚791022 disk.dm-1.in‚0 disk.sda.in‚0¸Žx’ ‚(:‡ ½×çÔñ»Áw gpu.0.gpu‚84.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ôr¡‚(:ç ½×çÔ¥ÚÍy disk.sda2.out‚0" proc.memory.rssMB‚ 3361.3203125* proc.memory.percent‚20.515871047973633 network.sent‚ 2.877298e+06 network.recv‚795521" disk.nvme0c0n1.in‚ 7422.6640625 disk.dm-0.in‚ 7422.6640625 disk.dm-1.out‚ 79.0546875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 7422.6640625# disk./.usageGB‚1156.60298538208( disk.nvme0c0n1.out‚58821.70654296875# disk.dm-0.out‚58821.70654296875 disk.dm-1.in‚0 disk.sda3.out‚ 79.0546875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚91.23833179473877) disk./.usagePercent‚77.62059680353498 disk.drbd1260.out‚ 0.35546875) proc.memory.availableMB‚ 1435.51171875 cpu‚24.754946511953644 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 79.0546875& disk.nvme0n1.out‚58821.70654296875 disk.sda2.in‚0Z¿:ß‘¢‚(:† Ì×çÔ±œ£y gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚42.0 gpu.0.powerWatts‚271.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m|‹ðo£‚(:ä Ì×çÔ‘ÉÄy( disk.nvme0c0n1.out‚59257.08154296875% disk./.usageGB‚1156.7267951965332 disk.drbd1260.out‚ 0.35546875 disk.sda2.out‚0 disk.sda1.out‚0# disk.dm-0.out‚59257.08154296875 disk.dm-1.out‚ 79.4453125 network.sent‚ 2.884065e+06 network.recv‚798969 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 79.4453125 disk.sda3.in‚0* proc.memory.percent‚20.515871047973633 proc.cpu.threads‚17) disk./.usagePercent‚77.62890578406628 disk.nvme0n1.in‚ 7422.6640625& disk.nvme0n1.out‚59257.08154296875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 79.4453125$ proc.memory.availableMB‚1324.875 disk.drbd1260.in‚0 disk.dm-0.in‚ 7422.6640625$ memory_percent‚91.91360473632812" proc.memory.rssMB‚ 3361.3203125 cpu‚24.757058912640325" disk.nvme0c0n1.in‚ 7422.66406255«â2¤- _wandbruntime‚2486  _runtime‚24867>LóÇ¥Á & lr-AdamW/pg2‚0.0006507102902850021 trainer/global_step‚1499 % _timestamp‚1.7884722797114108e+09 _runtime‚2486.467161577 _step‚119 & lr-AdamW/pg1‚0.0006507102902850021¬Žé7˦Š& lr-AdamW/pg1‚0.0006507102902850021 & lr-AdamW/pg2‚0.0006507102902850021 trainer/global_step‚1499 % _timestamp‚1.7884722797114108e+09 _runtime‚2486.467161577 _step‚119wAa2¨2§- _wandbruntime‚2486  _runtime‚2486ðôok=$¨öO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235490472910377 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241147406084096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3648648694272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33674656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231844585813872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241143769032704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123142144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37492580 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71939072 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 _runtime‚2486.468627711 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3642253005861 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37518918 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19986618 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69119555 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214982013060608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69145899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231844611241840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 % _timestamp‚1.7884722797125232e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13688735 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13688038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235486838819733 _step‚120 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚241147406084096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237495120338432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 _runtime‚2486.468627711 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3642253005861 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37518918 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2185 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4582277120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19986618 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69119555 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214982013060608 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69145899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231844611241840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 % _timestamp‚1.7884722797125232e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13688735 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13688038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33390854144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33390854144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235486838819733 _step‚120 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚241147406084096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237495120338432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241147406084096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3648648694272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33674656 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231844585813872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241143769032704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968x£ó>(2ª- _wandbruntime‚2488  _runtime‚2488Šû™i'«€N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3651059175936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237648924289024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31647405 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69164546 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3654669849088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31647399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5040321736192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 trainer/global_step‚1499 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33696209 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13697485 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5040393661952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚121 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚241303594138112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231994915055008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3648267972031 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3644659309147 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19999423 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69190916 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69190916 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237648897897472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3651059175936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33696913 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚241303594138112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚237648897897472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31647405 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220161748314624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235643208455007 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231994940482976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237648924289024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37543511 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 502894592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235639574364155 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37517147 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220161625185792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215121303449600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 % _timestamp‚1.7884722813741195e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912Ç°–& '¬„N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31647399 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220161625185792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215121303449600 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33390854144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4582277120 % _timestamp‚1.7884722813741195e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3651059175936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237648924289024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31647405 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69164546 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3654669849088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31647399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5040321736192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 trainer/global_step‚1499 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33696209 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13697485 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5040393661952 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚121 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚241303594138112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231994915055008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3648267972031 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3644659309147 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19999423 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69190916 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69190916 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237648897897472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3651059175936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33696913 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚241303594138112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚237648897897472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31647405 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220161748314624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3717 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235643208455007 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231994940482976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237648924289024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37543511 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 502894592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235639574364155 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4582277120 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37517147 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221946532510208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1549 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245184024681984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3769187191296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14131145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245183998290432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71372457 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38750620 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚248956822533120 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3769187191296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248953185481728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248953185481728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5203133517824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239361047870672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _runtime‚ 2561.2633385 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3766190424045 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20626873 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243127263722685 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38724282 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚731 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239361073298640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚227149666028032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38724282 % _timestamp‚1.7884723545077047e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221946583713280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14131871 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245183998290432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245184024681984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74036224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71398801 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38750620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 502894592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34758013 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0Å)bþ'¿øO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚227149791267328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243123629632041 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221946532510208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1549 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245184024681984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3769187191296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14131145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245183998290432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71372457 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38750620 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚248956822533120 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4582277120 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3769187191296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248953185481728 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248953185481728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4582277120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3717 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5203133517824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239361047870672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3717 _runtime‚ 2561.2633385 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3766190424045 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20626873 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243127263722685 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38724282 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚731 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239361073298640 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2185 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚227149666028032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38724282 % _timestamp‚1.7884723545077047e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221946583713280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33390854144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14131871 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3866051072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2185 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3717 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245183998290432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245184024681984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74036224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71398801 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38750620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 502894592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34758013 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20626868 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32648181 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125239296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34758744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248956822533120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3772797851136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6|¿eE2À- _wandbruntime‚2562  _runtime‚2562x¤ÓlÁ€N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71443818 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5206471569408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4582277120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245337775849472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239511402539776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243276365176463 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32668605 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 \JDeviceStatsMonitor.on_train_batcCŽ×žšh_end/requested_bytes.small_pool.allocated‚ 3768596727539 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚227292294468608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 502894592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32668599 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34779698 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20639678 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14140025 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71417448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71417448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222085822899200 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚239511377111808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245337802241024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3771597672960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14140735 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32668605 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38775213 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20639673 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34780413 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71443818 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚239511402539776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33390854144 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243276365176463 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32668605 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3768596727539 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚227292294468608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 502894592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32668599 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34779698 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20639678 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14140025 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71417448 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71417448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222085822899200 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚239511377111808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3717 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245337802241024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2185 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3771597672960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3866051072 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2185 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14140735 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32668605 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38775213 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20639673 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34780413 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71443818 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71443818 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884}Ë…2Ã- _wandbruntime‚2562  _runtime‚2562¹ 4ISÄÍ /loss/FRt3/assignment_loss‚3.953965738967613 1loss/FBt2/assignment_loss‚0.28002390037104347 /loss/FRt4/detection_loss‚2.5195477701896847 &loss/total_loss‚53.657804911990695 _step‚126 /loss/FRt4/assignment_loss‚4.449921887974407 epoch‚1 /loss/FRt2/assignment_loss‚3.332060031434323 2loss/SRqqt3/assignment_loss‚1.7340855362211787 0loss/SRqqt4/assignment_loss‚1.90403020059069 0loss/FBt3/assignment_loss‚0.5279524950776249 % _timestamp‚1.7884723555273852e+09 .loss/FRt2/detection_loss‚2.268036018026636 0loss/SRqqt1/detection_loss‚1.212610108104824 /loss/FBt2/detection_loss‚1.0992350231014913 .loss/FBt4/detection_loss‚2.764626044493455 /loss/FRt1/assignment_loss‚2.353608844936758 .loss/FRt3/detection_loss‚2.915920297304789 0loss/SRqqt3/detection_loss‚2.250658912629616 /loss/FBt3/detection_loss‚2.4765049695968626 2loss/FBt1/assignment_loss‚0.017326831972847383 1loss/FBt4/assignment_loss‚0.23245095167882168 /loss/SRqqt4/detection_loss‚2.41683047413826 _runtime‚2562.282693361 /loss/FRt1/detection_loss‚1.6990008940304413 1loss/SRqqt1/assignment_loss‚1.289678819148856 trainer/global_step‚1549 2loss/SRqqt2/assignment_loss‚2.4873424101443518 0loss/SRqqt2/detection_loss‚2.601937765166873 -loss/FBt1/detection_loss‚6.87044898668925“ÏaWÅÑ .loss/FBt4/detection_loss‚2.764626044493455 /loss/FRt1/assignment_loss‚2.353608844936758 .loss/FRt3/detection_loss‚2.915920297304789 0loss/SRqqt3/detection_loss‚2.250658912629616 /loss/FBt3/detection_loss‚2.4765049695968626 2loss/FBt1/assignment_loss‚0.017326831972847383 1loss/FBt4/assignment_loss‚0.23245095167882168 /loss/SRqqt4/detection_loss‚2.41683047413826 _runtime‚2562.282693361 /loss/FRt1/detection_loss‚1.6990008940304413 1loss/SRqqt1/assignment_loss‚1.289678819148856 trainer/global_step‚1549 2loss/SRqqt2/assignment_loss‚2.4873424101443518 0loss/SRqqt2/detection_loss‚2.601937765166873 -loss/FBt1/detection_loss‚6.87044898668925 /loss/FRt3/assignment_loss‚3.953965738967613 1loss/FBt2/assignment_loss‚0.28002390037104347 /loss/FRt4/detection_loss‚2.5195477701896847 &loss/total_loss‚53.657804911990695 _step‚126 /loss/FRt4/assignment_loss‚4.449921887974407 epoch‚1 /loss/FRt2/assignment_loss‚3.332060031434323 2loss/SRqqt3/assignment_loss‚1.7340855362211787 0loss/SRqqt4/assignment_loss‚1.90403020059069 0loss/FBt3/assignment_loss‚0.5279524950776249 % _timestamp‚1.7884723555273852e+09 .loss/FRt2/detection_loss‚2.268036018026636 0loss/SRqqt1/detection_loss‚1.212610108104824 /loss/FBt2/detection_loss‚1.0992350231014913~Z*ì½Æ‚(:’ ¦ØçÔ‚È…w gpu.0.gpu‚70.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.51667785644531* gpu.0.memoryAllocatedBytes‚ 33944567808 gpu.0.temp‚47.0 gpu.0.powerWatts‚258.061' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.612199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DRõkoÇ‚(:ä ¦ØçÔ‡Žãz disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1155.7351531982422 disk.dm-1.in‚0 disk.sda.in‚0 cpu‚24.771357276437875$ disk.nvme0n1.out‚61268.056640625 disk.sda.out‚ 82.0078125 disk.sda3.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 7694.15234375! disk.dm-0.out‚61268.056640625$ memory_percent‚95.82152366638184' proc.memory.availableMB‚ 684.6015625 network.sent‚ 3.019013e+06) disk./.usagePercent‚77.56235585751791 disk.sda2.out‚0& disk.nvme0c0n1.out‚61268.056640625 disk.dm-0.in‚ 7694.15234375# proc.memory.rssMB‚ 3361.32421875* proc.memory.percent‚20.515894889831543 proc.cpu.threads‚17 network.recv‚826313 disk.dm-1.out‚ 82.0078125 disk.sda3.out‚ 82.0078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3671875! disk.nvme0n1.in‚ 7694.15234375j÷â’È‚(:‡ µØçÔ¥‘™y gpu.0.gpu‚87.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚46.0 gpu.0.powerWatts‚278.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªS@vÉ‚(:ë µØçÔ•÷‘z disk.sda1.out‚0' disk.nvme0c0n1.out‚61268.6513671875$ memory_percent‚96.50352001190186 network.sent‚ 3.128167e+06 disk.sda.in‚0! disk.nvme0n1.in‚ 7694.15234375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 7694.15234375* proc.memory.percent‚20.515894889831543 cpu‚24.77302489638769 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-0.in‚ 7694.15234375" disk.dm-0.out‚61268.6513671875( proc.memory.availableMB‚ 572.86328125 network.recv‚833009 disk.dm-1.out‚ 82.33984375 disk.sda3.out‚ 82.33984375 disk.drbd1260.in‚0% disk.nvme0n1.out‚61268.6513671875 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3361.32421875) disk./.usagePercent‚77.56235457748025% disk./.usageGB‚1155.7351341247559 disk.sda.out‚ 82.33984375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.37109375VØ5o’Ê‚(:‡ ÄØçÔ¦¶†y gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚47.0 gpu.0.powerWatts‚272.011' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«à„sË‚(:è ÄØçÔÜ€þy% disk.nvme0n1.out‚61484.2138671875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚77.57077671334169% disk./.usageGB‚1155.8606300354004 disk.dm-1.in‚0 disk.sda3.out‚ 82.73046875 disk.sda1.out‚0( proc.memory.availableMB‚ 476.65234375! proc.memory.rssMB‚ 3361.328125* proc.memory.percent‚20.515918731689453 network.recv‚836201 disk.dm-0.in‚ 7694.15234375 disk.sda.out‚ 82.73046875 disk.drbd1260.out‚ 0.37109375 cpu‚24.774757567855346 network.sent‚ 3.132895e+06' disk.nvme0c0n1.out‚61484.2138671875 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 7694.15234375 disk.sda1.in‚0" memory_percent‚97.090744972229# disk.nvme0c0n1.in‚ 7694.15234375" disk.dm-0.out‚61484.2138671875 disk.dm-1.out‚ 82.73046875¯8‚GsÌ‚(:è ÓØçÔüÜÅx( disk.nvme0c0n1.out‚61845.22119140625 disk.dm-0.in‚ 7832.44921875) disk./.usagePercent‚77.56916360986844 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 7832.44921875( proc.memory.availableMB‚ 372.41015625 cpu‚24.777142113064343 disk.sda3.out‚ 83.18359375 disk.sda2.out‚0 disk.sda1.in‚0# disk.dm-0.out‚61845.22119140625 disk.dm-1.in‚0 disk.dm-1.out‚ 83.18359375! proc.memory.rssMB‚ 3361.328125 proc.cpu.threads‚17 network.recv‚840982 disk.sda.out‚ 83.18359375 disk.drbd1260.out‚0.375& disk.nvme0n1.out‚61845.22119140625 disk.sda1.out‚0$ memory_percent‚97.72698879241943* proc.memory.percent‚20.515918731689453 network.sent‚ 3.141186e+06% disk./.usageGB‚1155.8365936279297 disk.sda.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 7832.44921875 disk.sda2.in‚0v”±;’Í‚(:‡ ÓØçÔ¯©‰y gpu.0.gpu‚58.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚42.0 gpu.0.powerWatts‚210.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.0454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍœPï΂(:’ âØçÔ˜öÀw gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚43.0 gpu.0.powerWatts‚145.212' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.042399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>€ÔrÏ‚(:ç âØçÔó†¦x# disk.dm-0.out‚62231.18603515625 disk.sda3.out‚ 83.76953125 disk.drbd1260.in‚0& disk.nvme0n1.out‚62231.18603515625# memory_percent‚98.4121561050415# disk.nvme0c0n1.in‚ 7832.44921875( disk.nvme0c0n1.out‚62231.18603515625 disk.dm-1.out‚ 83.76953125 disk.sda3.in‚0 disk.drbd1260.out‚0.375 disk.sda2.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 3361.328125 network.sent‚ 3.143937e+06 disk.dm-0.in‚ 7832.44921875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 260.15234375* proc.memory.percent‚20.515918731689453 disk.sda.out‚ 83.76953125! disk.nvme0n1.in‚ 7832.44921875 disk.sda2.out‚0 cpu‚24.779202166694244 proc.cpu.threads‚17 network.recv‚843699) disk./.usagePercent‚77.57077517729648% disk./.usageGB‚1155.8606071472168TX(ï2Ð- _wandbruntime‚2625  _runtime‚2625ªýÁnÆÑÀ & lr-AdamW/pg1‚0.0006497880431992013 & lr-AdamW/pg2‚0.0006497880431992013 trainer/global_step‚1599 $ _timestamp‚1.788472418306014e+09 _runtime‚2625.061653983 _step‚127—DXÊÒÄ & lr-AdamW/pg1‚0.0006497880431992013 & lr-AdamW/pg2‚0.0006497880431992013 trainer/global_step‚1599 $ _timestamp‚1.788472418306014e+09 _runtime‚2625.061653983 _step‚127 /±¥2Ó- _wandbruntime‚2625  _runtime‚2625ñøÙÓÔöO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33669375 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39955984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14574765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74036224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2625.063314513 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3889725688320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39955984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252872876242432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39982322 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3879965184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39982322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5368982024704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33669381 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35841883 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4584374272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33669381 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252872902633984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252872876242432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33392951296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3893336348160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5369056060928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73625359 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2186 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 trainer/global_step‚1599 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35842608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125239296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250764054534993 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250760420444349 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4584374272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3882910516877 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33392951296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73651703 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228911051959808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3718 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3718 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3893336348160 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21267123 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33392951296 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256762601930752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228911103162880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246877509927472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3886519179553 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884724183074853e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234280159223808 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234280033984512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256766238982144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 517592064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2186…`l(ÕûO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39982322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5368982024704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33669381 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35841883 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4584374272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33669381 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252872902633984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252872876242432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33392951296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3893336348160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚720 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5369056060928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73625359 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2186 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 trainer/global_step‚1599 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35842608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125239296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250764054534993 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250760420444349 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4584374272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3882910516877 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33392951296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73651703 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228911051959808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3718 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3718 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3893336348160 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21267123 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33392951296 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256762601930752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚228911103162880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246877509927472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3886519179553 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884724183074853e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234280159223808 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234280033984512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256766238982144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 517592064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2186 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33669375 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39955984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14574765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74036224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2625.063314513 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3889725688320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39955984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252872876242432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39982322 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3879965184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚128€̪ºL2Ö- _wandbruntime‚2626  _runtime‚2626ü} '×ÿM BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39980551 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33689805 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35863611 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 517592064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3718 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3888925483047 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73696720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40006915 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247027864596576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33689805 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14583688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚256922427036160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4584374272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253026680193024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚256918789971456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3895746843136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33392951296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33689799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279923 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250916790079623 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229050393551872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234422635966464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Ê%µ|#Ø„N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229050342348800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚253026653801472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4584374272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3892136169984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2186 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39980551 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33689805 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35863611 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 517592064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3718 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3888925483047 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73696720 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40006915 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247027864596576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33689805 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14583688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚256922427036160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4584374272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253026680193024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚256918789971456 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3895746843136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33392951296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33689799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279923 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250916790079623 F=DeviceS·Ò½ŽtatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229050393551872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234422635966464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 Bs2é- _wandbruntime‚2705  _runtime‚2705ýê¯ü'êöO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15017823 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235875622612480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 517592064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41214024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235875571409408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34690575 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264572018379776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34690575 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36925917 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41187686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4010264185344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚721 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4006847935061 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264572018379776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34690581 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3718 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2186 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2186 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258397211256657 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4003239272385 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013874845184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264575655431168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241410520451072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34690581 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3718 _runtime‚2705.831475612 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264575655431168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36925191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254393997412240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33392951296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15018544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21907373 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013874845184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254393971984272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2186 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260561780585984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3879965184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4584374272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4010264185344’˜(ëûO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258397211256657 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4003239272385 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013874845184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264575655431168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241410520451072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34690581 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3718 _runtime‚2705.831475612 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264575655431168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36925191 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254393997412240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33392951296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15018544 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21907373 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013874845184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254393971984272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2186 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260561780585984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3879965184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4584374272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4010264185344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15017823 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235875622612480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 517592064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41214024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235875571409408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34690575 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264572018379776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34690575 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36925917 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41187686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4010264185344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚721 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4006847935061 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264572018379776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34690581 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3718 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2186 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2186 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840„öud2ì- _wandbruntime‚2708  _runtime‚2708î£Bôèí€N XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4012674667008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4584374272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5538336620032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36946816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5538260499968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.sêBrØ!mall_pool.freed‚ 4005645575671 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3718 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260715531753472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41212253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15027347 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264731843485184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3718 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258553580891931 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2186 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3879965184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚254544326653376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4584374272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚264728206420480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21920178 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4584374272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75923252 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258549946801079 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41212253 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4009254238555 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75949622 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241553122298368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2186 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 517592064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15026643 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2708.299705902 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21920173 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260715531753472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4016285340160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4012674667008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33392951296 trainer/global_step‚1649 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75923252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34710999 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260715558145024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952Öjq 'î…N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4012674667008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4584374272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5538336620032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36946816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5538260499968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4005645575671 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3718 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260715531753472 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41212253 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15027347 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264731843485184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3718 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258553580891931 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2186 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3879965184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚254544326653376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4584374272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚264728206420480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21920178 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4584374272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75923252 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258549946801079 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41212253 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4009254238555 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75949622 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241553122298368 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2186 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 517592064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15026643 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2708.299705902 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21920173 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260715531753472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4016285340160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4012674667008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33392951296 trainer/global_step‚1649 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75923252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34710999 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260715558145024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968…oì.2ï- _wandbruntime‚2708  _runtime‚2708éUðÏ .loss/FRt2/detection_loss‚2.304908224195242 1loss/SRqqt1/detection_loss‚1.1317918555794564 0loss/SRqqt2/detection_loss‚2.781296629654734 epoch‚1 _runtime‚2708.305526273 1loss/SRqqt3/assignment_loss‚1.958324084058404 /loss/FBt3/detection_loss‚2.4834642851352693 0loss/FRt1/assignment_loss‚2.2528999392923557 1loss/SRqqt2/assignment_loss‚2.222530998289585 0loss/FBt1/assignment_loss‚0.6511140973088914 /loss/FRt3/detection_loss‚3.1692057514190672 /loss/FBt1/detection_loss‚6.1273109912872314 0loss/FBt3/assignment_loss‚0.4903479097038507 .loss/FBt4/detection_loss‚2.875981399646172 1loss/SRqqt4/assignment_loss‚2.297882736722628 1loss/SRqqt3/detection_loss‚2.4070914406329393 0loss/SRqqt4/detection_loss‚2.595393576224645 /loss/FBt2/detection_loss‚0.9010714226269296 trainer/global_step‚1649 % _timestamp‚1.7884725015485084e+09 /loss/FRt2/assignment_loss‚2.894729767739773 0loss/FRt3/assignment_loss‚3.9357101273536683 /loss/FRt4/assignment_loss‚4.186420395093806 2loss/SRqqt1/assignment_loss‚1.2603606538171261 .loss/FRt1/detection_loss‚1.434855311109746 %loss/total_loss‚53.62751970790036 1loss/FBt2/assignment_loss‚0.28996069089262816 /loss/FBt4/assignment_loss‚0.303130015685593 /loss/FRt4/detection_loss‚2.6717374044306137 _step‚1345\UÊZñÔ .loss/FRt2/detection_loss‚2.304908224195242 1loss/SRqqt1/detection_loss‚1.1317918555794564 0loss/SRqqt2/detection_loss‚2.781296629654734 epoch‚1 _runtime‚2708.305526273 1loss/SRqqt3/assignment_loss‚1.958324084058404 /loss/FBt3/detection_loss‚2.4834642851352693 0loss/FRt1/assignment_loss‚2.2528999392923557 1loss/SRqqt2/assignment_loss‚2.222530998289585 0loss/FBt1/assignment_loss‚0.6511140973088914 /loss/FRt3/detection_loss‚3.1692057514190672 /loss/FBt1/detection_loss‚6.1273109912872314 0loss/FBt3/assignment_loss‚0.4903479097038507 .loss/FBt4/detection_loss‚2.875981399646172 1loss/SRqqt4/assignment_loss‚2.297882736722628 1loss/SRqqt3/detection_loss‚2.4070914406329393 0loss/SRqqt4/detection_loss‚2.595393576224645 /loss/FBt2/detection_loss‚0.9010714226269296 trainer/global_step‚1649 % _timestamp‚1.7884725015485084e+09 /loss/FRt2/assignment_loss‚2.894729767739773 0loss/FRt3/assignment_loss‚3.9357101273536683 /loss/FRt4/assignment_loss‚4.186420395093806 2loss/SRqqt1/assignment_loss‚1.2603606538171261 .loss/FRt1/detection_loss‚1.434855311109746 %loss/total_loss‚53.62751970790036 1loss/FBt2/assignment_loss‚0.28996069089262816 /loss/FBt4/assignment_loss‚0.303130015685593 /loss/FRt4/detection_loss‚2.6717374044306137 _step‚134†òË‘ò‚(:† ¼ÙçÔóÔµy gpu.0.gpu‚89.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚45.0 gpu.0.powerWatts‚266.675' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ”±ró‚(:ç ¼ÙçÔóz# disk.dm-0.out‚64179.95068359375% proc.memory.availableMB‚ 3.8828125 cpu‚24.792372423247915 proc.cpu.threads‚17% disk./.usageGB‚1157.0306777954102 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚64179.95068359375 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd1260.out‚0.390625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 7961.25390625 disk.dm-1.out‚ 86.58203125 disk.sda.out‚ 86.58203125 network.sent‚ 3.279638e+06 disk.sda3.out‚ 86.58203125! disk.nvme0n1.in‚ 7961.25390625& disk.nvme0n1.out‚64179.95068359375 disk.sda2.in‚0# memory_percent‚99.9763011932373" proc.memory.rssMB‚ 3361.3359375* proc.memory.percent‚20.515966415405273 network.recv‚870432( disk./.usagePercent‚77.6492996002514 disk.sda3.in‚0 disk.dm-0.in‚ 7961.25390625Wkt’ô‚(:‡ ËÙçÔùÉx gpu.0.gpu‚53.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚41.0 gpu.0.powerWatts‚196.283' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.2566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜX 9oõ‚(:ä ËÙçÔ¦¾æy" proc.memory.rssMB‚ 3361.3359375 cpu‚24.79489781274912 proc.cpu.threads‚17 network.recv‚877898% disk./.usageGB‚1157.0515937805176# disk.nvme0c0n1.in‚ 7989.03515625" disk.dm-0.out‚64599.4580078125 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 7989.03515625 disk.sda1.out‚0* proc.memory.percent‚20.515966415405273' disk.nvme0c0n1.out‚64599.4580078125 disk.sda3.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 6.7890625 disk.sda3.out‚ 87.0390625 network.sent‚ 3.389218e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 87.0390625 disk.sda.out‚ 87.0390625% disk.nvme0n1.out‚64599.4580078125 disk.sda1.in‚0$ memory_percent‚99.95856285095215) disk./.usagePercent‚77.65070328956163 disk.dm-0.in‚ 7989.03515625 disk.sda.in‚0 disk.drbd1260.out‚ 0.39453125 disk.sda2.in‚0<¯¼‘ö‚(:† ÚÙçÔãœñw gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚44.0 gpu.0.powerWatts‚282.254' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.4508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç¦À[d÷‚(:Ù ÚÙçÔΨçx network.sent‚ 3.396111e+06) disk./.usagePercent‚77.66748227936328$ disk./.usageGB‚1157.301612854004 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.39453125 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 8077.87890625 disk.dm-0.in‚ 8077.87890625 disk.sda.out‚87.46875% disk.nvme0n1.out‚65053.3525390625' disk.nvme0c0n1.out‚65053.3525390625$ memory_percent‚99.96795654296875 proc.memory.availableMB‚5.25" proc.memory.rssMB‚ 3361.3359375* proc.memory.percent‚20.515966415405273 proc.cpu.threads‚17! disk.nvme0n1.in‚ 8077.87890625 network.recv‚881435 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚24.797230352291564 disk.dm-1.out‚87.46875 disk.sda.in‚0 disk.sda3.out‚87.46875 disk.sda2.in‚0" disk.dm-0.out‚65053.3525390625.›%Koø‚(:ä éÙçÔ߇¹x disk.dm-1.out‚ 87.88671875 disk.sda.out‚ 87.88671875 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 8077.87890625 disk.sda2.in‚0( disk./.usagePercent‚77.6674809993256# disk.dm-0.out‚65207.52490234375 disk.sda3.out‚ 87.88671875 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚99.95880126953125" proc.memory.rssMB‚ 3361.3359375* proc.memory.percent‚20.515966415405273 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.3984375 disk.sda1.in‚0 network.sent‚ 3.400318e+06 network.recv‚886161 disk.dm-0.in‚ 8077.87890625 disk.sda.in‚0& disk.nvme0n1.out‚65207.52490234375 proc.memory.availableMB‚6.75 cpu‚24.799439881400204 proc.cpu.threads‚17% disk./.usageGB‚1157.3015937805176# disk.nvme0c0n1.in‚ 8077.87890625( disk.nvme0c0n1.out‚65207.52490234375®e´j’ù‚(:‡ éÙçÔÀ…»x gpu.0.gpu‚63.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚42.0 gpu.0.powerWatts‚214.016' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.8032 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒíeoú‚(:ä øÙçÔ‡è¾x( disk.nvme0c0n1.out‚65541.31396484375# disk.dm-0.out‚65541.31396484375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 3.40517e+06) disk./.usagePercent‚77.66748227936328# disk.nvme0c0n1.in‚ 8077.87890625 disk.dm-0.in‚ 8077.87890625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 88.3359375 disk.drbd1260.out‚ 0.3984375 disk.sda.out‚ 88.3359375! disk.nvme0n1.in‚ 8077.87890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 3.8515625" proc.memory.rssMB‚ 3361.3359375 cpu‚24.80188017726053$ disk./.usageGB‚1157.301612854004 disk.dm-1.out‚ 88.3359375& disk.nvme0n1.out‚65541.31396484375$ memory_percent‚99.97649192810059* proc.memory.percent‚20.515966415405273 network.recv‚889567žót7’û‚(:‡ øÙçÔâÇ­y gpu.0.gpu‚56.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚41.0 gpu.0.powerWatts‚217.688' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.5376 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒï4ñNü‚(:à ‡ÚçÔ´¹¿y! disk.nvme0n1.in‚ 8077.87890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 8077.87890625 disk.dm-0.in‚ 8077.87890625 disk.dm-0.out‚65840.74609375 disk.dm-1.out‚88.75 network.sent‚ 3.413381e+06 network.recv‚894181) disk./.usagePercent‚77.70103641885355% disk./.usageGB‚1157.8015937805176 disk.sda1.in‚0% disk.nvme0c0n1.out‚65840.74609375 disk.dm-1.in‚0 disk.sda.in‚0# disk.nvme0n1.out‚65840.74609375 disk.sda.out‚88.75 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.40234375" proc.memory.rssMB‚ 3361.3359375 disk.sda3.out‚88.75 memory_percent‚ 99.9755859375 proc.memory.availableMB‚4* proc.memory.percent‚20.515966415405273 cpu‚24.80410480331683 proc.cpu.threads‚17¦Îfý‚(:’ ‡ÚçÔªñ”z gpu.0.gpu‚74.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚40.0 gpu.0.powerWatts‚183.152' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.630399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðÓ2þ- _wandbruntime‚2791  _runtime‚2791Nz7ÇÿÁ & lr-AdamW/pg1‚0.0006485958407625374 & lr-AdamW/pg2‚0.0006485958407625374 trainer/global_step‚1699 % _timestamp‚1.7884725851994011e+09 _runtime‚2791.955215112 _step‚135{(ÈÅÌ€Æ & lr-AdamW/pg1‚0.0006485958407625374 & lr-AdamW/pg2‚0.0006485958407625374 trainer/global_step‚1699 % _timestamp‚1.7884725851994011e+09 _runtime‚2791.955215112 _step‚135‡„Æþ2- _wandbruntime‚2791  _runtime‚27917Ž‰Ä ‚öO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35711781 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5700833252864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4130802682368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268250658537984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚729 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3718 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78131163 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚242840090859008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35711775 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42445726 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33392951296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35711781 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268250632146432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268250658537984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38010725 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38009991 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2186 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272385071880192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272381434828800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42445726 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272385071880192 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15462373 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261910459469040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33392951296 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 M@DeviceStatsMonitor.on_trø뛌}ain_batch_start/allocation.all.allocated‚78157507 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4123568027893 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33392951296 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884725852004411e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261910434041072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2186 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42419388 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4584374272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2791.956618728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4584374272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272381434828800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42419388 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4134413342208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952µ¡ê~(ƒûO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38009991 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2186 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272385071880192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272381434828800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42445726 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272385071880192 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15462373 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚261910459469040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33392951296 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78157507 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4123568027893 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33392951296 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884725852004411e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261910434041072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2186 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42419388 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4584374272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3718 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2791.956618728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4584374272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272381434828800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42419388 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4134413342208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35711781 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5700833252864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4130802682368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268250658537984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚729 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3718 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78131163 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚242840090859008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35711775 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42445726 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33392951296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35711781 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268250632146432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268250658537984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38010725ˆ/ ­2„- _wandbruntime‚2793  _runtime‚2793¦%Ÿ´'…ÿM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3718 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4584374272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3879965184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4125974331179 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4136823837184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788472586763005e+09 _runtime‚2793.523463248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78202524 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38031697 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248683643307520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266186737613387 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚137 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268404409705472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78176154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35732205 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248683518081536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4129582994063 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268404409705472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22560423 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚242979432451072 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2186 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38031697 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚248683643307520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266186737613387 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚137 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268404409705472 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78176154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35732205 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248683518081536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4129582994063 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268404409705472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22560423 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚242979432451072 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2186 Bâ.peak‚2050 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4136823837184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788472586763005e+09 _runtime‚2793.523463248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78202524 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36732981 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 517592064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4243896783401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36732975 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23187868 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚280194488329216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4254951839232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4584374272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4251341179392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33392951296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 _step‚140 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43651090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15906586 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273670792881273 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43677428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125239296 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚280190851277824µAàÌ™úO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2050 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15905871 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2186 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23187873 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255671435640832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269426896097872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4584374272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3879965184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚275939510098432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33392951296 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249804661511680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚280190851277824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36732981 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 517592064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4243896783401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36732975 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23187868 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚280194488329216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4254951839232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4584374272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4251341179392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33392951296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 _step‚140 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43651090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15906586 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚273670792881273 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43677428 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125239296 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39115355 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4253751661056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249943900697600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269577225339008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4584374272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚273823528425695 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80455426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15914682 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280347039318528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3718 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2186 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276093314049024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4257362334208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4246303086687 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884726728054814e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3718 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255814051941888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80429056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276093287657472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276093314049024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255813926715904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276093287657472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4584374272 trainer/global_step‚1749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80455426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚695 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15914682 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280347039318528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3718 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2186 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276093314049024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4257362334208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4246303086687 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884726728054814e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3718 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255814051941888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80429056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276093287657472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276093314049024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255813926715904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚276093287657472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4584374272 trainer/global_step‚1749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39115355 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4253751661056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚249943900697600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269577225339008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4584374272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚273823528425695 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370<£çž2- _wandbruntime‚2879  _runtime‚2879‘´ZžÔ 1loss/SRqqt3/detection_loss‚2.3430333980198563 1loss/SRqqt4/detection_loss‚2.8851332235336304 2loss/SRqqt3/assignment_loss‚1.9121138774629296 /loss/FRt1/detection_loss‚1.5877089459667857 -loss/FRt3/detection_loss‚3.20371271421512 /loss/FBt2/detection_loss‚0.9877430949303688 .loss/FBt4/detection_loss‚2.816287730421339 % _timestamp‚1.7884726728078332e+09 0loss/FRt3/assignment_loss‚3.4097927610079446 0loss/FBt2/assignment_loss‚0.2752219276527383 1loss/FBt3/assignment_loss‚0.41002930930907106 0loss/FBt4/assignment_loss‚0.7238004367266377 _runtime‚2879.563159292 /loss/FRt2/assignment_loss‚3.227232734362284 0loss/FRt4/assignment_loss‚3.7856043001701094 2loss/FBt1/assignment_loss‚0.010387381911277771 &loss/total_loss‚50.934757826021524 2loss/SRqqt4/assignment_loss‚1.9087101715803145 1loss/SRqqt1/detection_loss‚1.0386008035979772 epoch‚1 _step‚142 0loss/FRt1/assignment_loss‚2.2894811860050064 2loss/SRqqt2/assignment_loss‚1.7462339659531911 -loss/FRt2/detection_loss‚2.13600277350772 /loss/FBt1/detection_loss‚4.6901536107063295 trainer/global_step‚1749 1loss/SRqqt1/assignment_loss‚1.240025747301323 1loss/SRqqt2/detection_loss‚2.9446480671564736 .loss/FBt3/detection_loss‚2.421607645494597 .loss/FRt4/detection_loss‚2.941492019028499»a‹D_ŸÙ .loss/FRt4/detection_loss‚2.941492019028499 1loss/SRqqt3/detection_loss‚2.3430333980198563 1loss/SRqqt4/detection_loss‚2.8851332235336304 2loss/SRqqt3/assignment_loss‚1.9121138774629296 /loss/FRt1/detection_loss‚1.5877089459667857 -loss/FRt3/detection_loss‚3.20371271421512 /loss/FBt2/detection_loss‚0.9877430949303688 .loss/FBt4/detection_loss‚2.816287730421339 % _timestamp‚1.7884726728078332e+09 0loss/FRt3/assignment_loss‚3.4097927610079446 0loss/FBt2/assignment_loss‚0.2752219276527383 1loss/FBt3/assignment_loss‚0.41002930930907106 0loss/FBt4/assignment_loss‚0.7238004367266377 _runtime‚2879.563159292 /loss/FRt2/assignment_loss‚3.227232734362284 0loss/FRt4/assignment_loss‚3.7856043001701094 2loss/FBt1/assignment_loss‚0.010387381911277771 &loss/total_loss‚50.934757826021524 2loss/SRqqt4/assignment_loss‚1.9087101715803145 1loss/SRqqt1/detection_loss‚1.0386008035979772 epoch‚1 _step‚142 0loss/FRt1/assignment_loss‚2.2894811860050064 2loss/SRqqt2/assignment_loss‚1.7462339659531911 -loss/FRt2/detection_loss‚2.13600277350772 /loss/FBt1/detection_loss‚4.6901536107063295 trainer/global_step‚1749 1loss/SRqqt1/assignment_loss‚1.240025747301323 1loss/SRqqt2/detection_loss‚2.9446480671564736 .loss/FBt3/detection_loss‚2.421607645494597Ž«D<ýe ‚(:Ú áÚçÔÖλx disk.dm-1.out‚ 91.0234375 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3361.3359375* proc.memory.percent‚20.515966415405273 proc.cpu.threads‚17% disk./.usageGB‚1157.7782440185547( disk.nvme0c0n1.out‚68114.65185546875# disk.dm-0.out‚68114.65185546875 disk.sda.out‚ 91.0234375 disk.sda3.in‚0 disk.nvme0n1.in‚ 8363.8125 disk.sda2.out‚0$ memory_percent‚99.96676445007324 network.sent‚ 3.547841e+06 network.recv‚920438) disk./.usagePercent‚77.69946939673648 disk.nvme0c0n1.in‚ 8363.8125 disk.dm-0.in‚ 8363.8125 disk.dm-1.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 5.4453125 disk.sda.in‚0 disk.sda3.out‚ 91.0234375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4140625& disk.nvme0n1.out‚68114.65185546875 cpu‚24.81701887014037&ÔÍd¡‚(:’ áÚçÔÚŒÔx gpu.0.gpu‚51.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚41.0 gpu.0.powerWatts‚132.338' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.467599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÎ墂(:… ðÚçÔå©šy gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J£ðj£‚(:ß ðÚçÔº¯y cpu‚24.819091716506062 network.sent‚ 3.655728e+06 network.recv‚925882 disk.dm-1.out‚ 91.38671875 disk.sda.out‚ 91.38671875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 8363.8125) disk./.usagePercent‚77.70104589113232 disk.nvme0c0n1.in‚ 8363.8125 disk.dm-0.in‚ 8363.8125# disk.dm-0.out‚68558.08544921875 disk.sda2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1157.8017349243164 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.4140625 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚68558.08544921875 disk.sda3.out‚ 91.38671875& disk.nvme0n1.out‚68558.08544921875$ memory_percent‚99.97541904449463& proc.memory.availableMB‚ 4.02734375" proc.memory.rssMB‚ 3361.3359375* proc.memory.percent‚20.515966415405273'?]hœ¤‚(:‘ ÿÚçÔú·æx gpu.0.gpu‚88.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.265' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.65299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%Võ±j¥‚(:ß ÿÚçÔÇù z disk.dm-1.in‚0 disk.dm-1.out‚ 91.83984375 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.96190071105957 network.recv‚930616 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚68868.61376953125 disk.sda2.out‚0* proc.memory.percent‚20.515966415405273 cpu‚24.821309147955155 network.sent‚ 3.664001e+06 disk.nvme0c0n1.in‚ 8363.8125( disk.nvme0c0n1.out‚68868.61376953125# disk.dm-0.out‚68868.61376953125 disk.nvme0n1.in‚ 8363.8125) disk./.usagePercent‚77.70104614713986% disk./.usageGB‚1157.8017387390137 disk.sda.out‚ 91.83984375 disk.sda3.out‚ 91.83984375 disk.drbd1260.out‚ 0.41796875% proc.memory.availableMB‚ 6.2421875" proc.memory.rssMB‚ 3361.3359375 proc.cpu.threads‚17 disk.dm-0.in‚ 8363.8125Í‹YUq¦‚(:æ ŽÛçÔóÜËx disk.sda.out‚ 92.20703125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.41796875% disk.nvme0n1.out‚69312.0693359375$ proc.memory.availableMB‚5.796875" proc.memory.rssMB‚ 3361.3359375 cpu‚24.82338359935859' disk.nvme0c0n1.out‚69312.0693359375 disk.dm-0.in‚ 8480.49609375 network.sent‚ 3.666965e+06% disk./.usageGB‚1157.8015441894531 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 8480.49609375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.out‚ 92.20703125 disk.drbd1260.in‚0$ memory_percent‚99.96461868286133* proc.memory.percent‚20.515966415405273 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 92.20703125 network.recv‚933849) disk./.usagePercent‚77.70103309075562! disk.nvme0n1.in‚ 8480.49609375" disk.dm-0.out‚69312.0693359375Åæç‚(:’ ŽÛçԆݟy gpu.0.gpu‚70.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚40.0 gpu.0.powerWatts‚180.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.064800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚TÓ㨂(:’ ÛçÔ¾ó¼w gpu.0.gpu‚88.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.599599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿáçét©‚(:é ÛçÔôìÃx# disk.nvme0c0n1.in‚ 8480.49609375( disk.nvme0c0n1.out‚69466.16259765625# disk.dm-0.out‚69466.16259765625 disk.sda.in‚0 disk.sda3.out‚ 92.70703125 disk.drbd1260.out‚0.421875 disk.dm-0.in‚ 8480.49609375 disk.dm-1.out‚ 92.70703125! disk.nvme0n1.in‚ 8480.49609375& disk.nvme0n1.out‚69466.16259765625$ memory_percent‚99.96113777160645% proc.memory.availableMB‚ 6.3671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 92.70703125 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚20.515990257263184 cpu‚24.82516267374875 network.recv‚938354% disk./.usageGB‚1156.5579643249512 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3361.33984375 proc.cpu.threads‚17 network.sent‚ 3.673178e+06) disk./.usagePercent‚77.61757540261588Äþ2ª- _wandbruntime‚2954  _runtime‚2954~ßãdÄ«¾ _step‚143 % lr-AdamW/pg1‚0.000647134676736037 % lr-AdamW/pg2‚0.000647134676736037 trainer/global_step‚1799 % _timestamp‚1.7884727473569345e+09 _runtime‚ 2954.11251082»ó£ ɬà % lr-AdamW/pg2‚0.000647134676736037 trainer/global_step‚1799 % _timestamp‚1.7884727473569345e+09 _runtime‚ 2954.11251082 _step‚143 % lr-AdamW/pg1‚0.000647134676736037âÖ®ä2­- _wandbruntime‚2954  _runtime‚2954~”Ïtò®öO _runtime‚2954.114168475 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288000267726848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16348975 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4584374272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283628388050432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3879965184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283628388050432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4375490336256 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2186 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6032737753600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33392951296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4375490336256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283628414441984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44882792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16349688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3718 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82663311 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚276943358154672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4367834201585 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281307583693581 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37754181 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 % _timestamp‚1.7884727473581874e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44909130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44882792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262801992751104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37754181 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256769129758208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3718 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37754175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44909130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4371879676416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33392951296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288003904778240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40177811 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23828123 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256769180961280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283628414441984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23828118 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74036224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262801867511808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4364225538909 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ŒÞ¢™(¯ûO % _timestamp‚1.7884727473581874e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44909130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44882792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262801992751104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37754181 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256769129758208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3718 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37754175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44909130 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4371879676416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33392951296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288003904778240 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40177811 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23828123 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256769180961280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283628414441984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23828118 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74036224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262801867511808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4364225538909 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4584374272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2954.114168475 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288000267726848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16348975 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1181 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4584374272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283628388050432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3879965184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283628388050432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4375490336256 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2186 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6032737753600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33392951296 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4375490336256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283628414441984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44882792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16349688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3718 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82663311 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚276943358154672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4367834201585 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281307583693581 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1799 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37754181 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877¥J4+q°‚(:æ ¬ÛçÔµá…y disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 8480.49609375 disk.dm-1.in‚0 network.recv‚941703$ disk./.usageGB‚1155.653678894043 disk.sda3.out‚ 93.03515625! disk.nvme0n1.in‚ 8480.49609375 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 8480.49609375# disk.dm-0.out‚70072.77978515625 disk.sda.in‚0 disk.sda.out‚ 93.03515625$ memory_percent‚99.95906352996826& proc.memory.availableMB‚ 6.70703125 proc.memory.rssMB‚ 3361.34375 proc.cpu.threads‚17( disk./.usagePercent‚77.5568880486016 disk.drbd1260.out‚0.421875& disk.nvme0n1.out‚70072.77978515625 disk.sda2.out‚0( disk.nvme0c0n1.out‚70072.77978515625 disk.dm-1.out‚ 93.03515625* proc.memory.percent‚20.516014099121094 cpu‚24.826423385358986 network.sent‚ 3.678055e+06T»&±‚(:’ ¬ÛçÔ¥Çy gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚47.0 gpu.0.powerWatts‚274.378' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.875600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª­üÕ2²- _wandbruntime‚2955  _runtime‚2955³t\'³€N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3879965184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚281463953328855 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33392951296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288156455767552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82681958 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3718 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4584374272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288160092832256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16358502 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40198721 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256908471350272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82708328 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44933723 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6036128445952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3718 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44907359 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16357798 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37774605 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 517592064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262944472473088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3718 _runtime‚2955.156658402 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚283782165609472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4377900831232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262944599796224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚283782165609472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23840928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277093712823776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4370240505079 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4584374272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3718 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23840923 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6036052325888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4377900831232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37774605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4366631842195 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2186 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4584374272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4374290158080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33392951296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33392951296 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884727483998837e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44933723 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288160092832256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281460319238003 trainer/global_step‚1799 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277093687395808 _step‚145 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884727483998837e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool`Eo .peak‚ 4509786112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44933723 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288160092832256 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281460319238003 trainer/global_step‚1799 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277093687395808 _step‚145 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40198721 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256908471350272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82708328 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2050 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44933723 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6036128445952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3718 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44907359 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16357798 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37774605 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 517592064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262944472473088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3718 _runtime‚2955.156658402 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚283782165609472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4377900831232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262944599796224 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚283782165609472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23840928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277093712823776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4370240505079 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4584374272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3718 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23840923 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1181 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6036052325888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4377900831232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37774605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4366631842195 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2186‘Ql–Ï2µ- _wandbruntime‚2955  _runtime‚2955 ØY¶Ó 2loss/SRqqt1/assignment_loss‚1.2507099696834172 /loss/FRt4/detection_loss‚2.6617784783607576 2loss/SRqqt3/assignment_loss‚2.1629919177469086 2loss/SRqqt4/assignment_loss‚2.2600972030697197 1loss/FBt4/assignment_loss‚0.13357793822976305 /loss/FRt3/detection_loss‚3.4038871356419156 1loss/SRqqt1/detection_loss‚1.1524413511423128 _runtime‚ 2955.15865177 _step‚146 0loss/FRt1/assignment_loss‚2.7653331522700153 0loss/FRt2/assignment_loss‚2.6054732015258386 /loss/FBt2/detection_loss‚0.8725329532564895 %loss/total_loss‚52.66722551315899 0loss/FBt3/assignment_loss‚0.5804384498512193 /loss/FRt2/detection_loss‚2.4626276699924157 -loss/FBt1/detection_loss‚4.21375298500061 $ _timestamp‚1.788472748403163e+09 /loss/FRt3/assignment_loss‚4.606660933721633 2loss/SRqqt2/assignment_loss‚2.5235020043673337 .loss/FBt3/detection_loss‚1.994097426533699 trainer/global_step‚1799 /loss/FRt1/detection_loss‚1.5511792893521488 1loss/SRqqt2/detection_loss‚2.5033010553430626 .loss/FBt4/detection_loss‚3.047216236591339 epoch‚1 /loss/FRt4/assignment_loss‚4.685417657945214 1loss/FBt1/assignment_loss‚0.19365474283695222 0loss/FBt2/assignment_loss‚0.2617416774204657 1loss/SRqqt3/detection_loss‚2.1877806099022137 1loss/SRqqt4/detection_loss‚2.5870314733735444Ò>#-^·Ø 2loss/SRqqt2/assignment_loss‚2.5235020043673337 .loss/FBt3/detection_loss‚1.994097426533699 trainer/global_step‚1799 /loss/FRt1/detection_loss‚1.5511792893521488 1loss/SRqqt2/detection_loss‚2.5033010553430626 .loss/FBt4/detection_loss‚3.047216236591339 epoch‚1 /loss/FRt4/assignment_loss‚4.685417657945214 1loss/FBt1/assignment_loss‚0.19365474283695222 0loss/FBt2/assignment_loss‚0.2617416774204657 1loss/SRqqt3/detection_loss‚2.1877806099022137 1loss/SRqqt4/detection_loss‚2.5870314733735444 2loss/SRqqt1/assignment_loss‚1.2507099696834172 /loss/FRt4/detection_loss‚2.6617784783607576 2loss/SRqqt3/assignment_loss‚2.1629919177469086 2loss/SRqqt4/assignment_loss‚2.2600972030697197 1loss/FBt4/assignment_loss‚0.13357793822976305 /loss/FRt3/detection_loss‚3.4038871356419156 1loss/SRqqt1/detection_loss‚1.1524413511423128 _runtime‚ 2955.15865177 _step‚146 0loss/FRt1/assignment_loss‚2.7653331522700153 0loss/FRt2/assignment_loss‚2.6054732015258386 /loss/FBt2/detection_loss‚0.8725329532564895 %loss/total_loss‚52.66722551315899 0loss/FBt3/assignment_loss‚0.5804384498512193 /loss/FRt2/detection_loss‚2.4626276699924157 -loss/FBt1/detection_loss‚4.21375298500061 $ _timestamp‚1.788472748403163e+09 /loss/FRt3/assignment_loss‚4.606660933721633’]ÈXp¸‚(:å »ÛçÔ¦þ÷x network.recv‚948410 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.42578125& disk.nvme0n1.out‚70362.82666015625 disk.sda2.out‚0 disk.dm-0.in‚ 8480.49609375 disk.dm-1.out‚ 93.5234375 cpu‚24.827682048165606 network.sent‚ 3.775254e+06% disk./.usageGB‚1155.6536827087402 disk.sda3.out‚ 93.5234375 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.in‚0$ memory_percent‚99.95927810668945$ proc.memory.availableMB‚6.671875) disk./.usagePercent‚77.55688830460915! disk.nvme0n1.in‚ 8480.49609375 disk.dm-1.in‚0 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 8480.49609375( disk.nvme0c0n1.out‚70362.82666015625# disk.dm-0.out‚70362.82666015625 disk.sda.out‚ 93.5234375—žF’¹‚(:‡ »ÛçÔ¢Ÿ„y gpu.0.gpu‚74.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚39.51911926269531* gpu.0.memoryAllocatedBytes‚ 33946664960 gpu.0.temp‚46.0 gpu.0.powerWatts‚275.519' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1038 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊgÆÊœº‚(:‘ ÊÛçÔð¥Íw gpu.0.gpu‚73.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚278.938' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.78759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®mR_o»‚(:ä ÊÛçÔ’ŒÃx disk.sda2.out‚0 disk.sda1.in‚0 cpu‚24.8290146089735 network.sent‚ 3.872406e+06 disk.sda3.in‚0 disk.sda3.out‚ 93.8203125 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚77.55688625654888$ disk./.usageGB‚1155.653652191162# disk.dm-0.out‚70820.86181640625 disk.dm-1.out‚ 93.8203125 disk.sda.in‚0 disk.sda.out‚ 93.8203125 disk.drbd1260.out‚ 0.42578125! disk.nvme0n1.in‚ 8596.42578125 network.recv‚953657# disk.nvme0c0n1.in‚ 8596.42578125 disk.dm-0.in‚ 8596.42578125 disk.sda1.out‚0$ memory_percent‚99.95925426483154& proc.memory.availableMB‚ 6.67578125 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094( disk.nvme0c0n1.out‚70820.86181640625 disk.dm-1.in‚0 disk.drbd1260.in‚0& disk.nvme0n1.out‚70820.86181640625Ž+ºp¼‚(:å ÙÛçÔðï‰y disk.drbd1260.out‚ 0.42578125 proc.memory.rssMB‚ 3361.34375 network.recv‚958225 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-0.in‚ 8596.42578125 disk.dm-1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚20.516014099121094 network.sent‚ 3.878407e+06! disk.nvme0n1.in‚ 8596.42578125& disk.nvme0n1.out‚70893.62255859375# disk.dm-0.out‚70893.62255859375# memory_percent‚99.9577283859253& proc.memory.availableMB‚ 6.92578125 cpu‚24.831287812910652 proc.cpu.threads‚17 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 8596.42578125( disk.nvme0c0n1.out‚70893.62255859375 disk.sda.out‚ 94.3984375 disk.sda3.in‚0) disk./.usagePercent‚77.55688139240573$ disk./.usageGB‚1155.653579711914 disk.sda1.in‚0 disk.dm-1.out‚ 94.3984375 disk.sda.in‚0 disk.sda3.out‚ 94.3984375Ð};h‘½‚(:† ÙÛçÔºÝñy gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚184.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý7ip¾‚(:å èÛçÔÆèz) disk./.usagePercent‚77.55031454314492( disk.nvme0c0n1.out‚71019.61572265625 disk.drbd1260.in‚0& disk.nvme0n1.out‚71019.61572265625 disk.sda2.in‚0 disk.sda1.in‚0# memory_percent‚99.9608039855957 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 cpu‚24.83255451408098# disk.nvme0c0n1.in‚ 8596.42578125 disk.dm-0.in‚ 8596.42578125# disk.dm-0.out‚71019.61572265625 disk.dm-1.in‚0 disk.dm-1.out‚ 94.82421875 disk.sda.out‚ 94.82421875 disk.sda3.out‚ 94.82421875 disk.drbd1260.out‚ 0.4296875 network.sent‚ 3.883383e+06 network.recv‚961754% disk./.usageGB‚1155.5557289123535! disk.nvme0n1.in‚ 8596.42578125$ proc.memory.availableMB‚6.421875 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17§oÐ)’¿‚(:‡ èÛçÔÜÀÅz gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚245.296' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.0592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m£àQ2À- _wandbruntime‚3018  _runtime‚3018Ôë±}ÇÁÁ & lr-AdamW/pg1‚0.0006463036060062022 & lr-AdamW/pg2‚0.0006463036060062022 trainer/global_step‚1849 % _timestamp‚1.7884728116554568e+09 _runtime‚3018.411028831 _step‚147,KÚÌÂÆ _runtime‚3018.411028831 _step‚147 & lr-AdamW/pg1‚0.0006463036060062022 & lr-AdamW/pg2‚0.0006463036060062022 trainer/global_step‚1849 % _timestamp‚1.7884728116554568e+09“Ôy2Ã- _wandbruntime‚3018  _runtime‚3018àðUñú'ÄôO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2058 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269932185890816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288948008596533 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84889869 _runtime‚ 3018.41207292 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41260156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚284459820211472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4492418173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291317266002432 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46140832 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4484554294417 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 _step‚148 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4492418173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38775381 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263733700410880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1189 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288944374505889 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295813321227264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84916213 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295809684175872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6198485479936 trainer/global_step‚1849 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4492418173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291317266002432 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46140832 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4484554294417 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 _step‚148 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4492418173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38775381 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263733700410880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1189 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288944374505889 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295813321227264 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84916213 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295809684175872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6198485479936 trainer/global_step‚1849 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4498439328256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84961230 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263872990799872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38795805 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291471069953024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41282466 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84961230 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291471043561472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16800578 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4490569260587 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4494828655104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2058 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289097110050311 trainer/global_step‚1849 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84934860 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291471069953024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46139061 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46165425 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3886256640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚284610174880576 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4486960597703 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46165425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46139061 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84934860 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263872939596800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6201745651200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38795805 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚270074685248000 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295969509281280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41281751 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚270074812571136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4494828655104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚3019.468952802 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚295965872216576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24481173 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6201821771264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24481178 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291471043561472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16801288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38795799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚295969509281280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0cÝý? 'È…N REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46165425 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3886256640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚284610174880576 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4486960597703 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46165425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46139061 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84934860 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263872939596800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6201745651200 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38795805 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚270074685248000 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295969509281280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41281751 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚270074812571136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4494828655104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚3019.468952802 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚295965872216576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24481173 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6201821771264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24481178 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291471043561472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16801288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38795799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚295969509281280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1189 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 % _timestamp‚1.7884728127124596e+09 _step‚149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38795799 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 523944960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4498439328256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84961230 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263872990799872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38795805 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291471069953024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41282466 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84961230 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291471043561472 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16800578 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4490569260587 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4494828655104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2058 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚289097110050311 trainer/global_step‚1849 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84934860 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291471069953024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46139061•°ê?m2É- _wandbruntime‚3019  _runtime‚3019S‹š­WÊÑ /loss/FRt3/assignment_loss‚3.663824955622355 2loss/SRqqt3/assignment_loss‚2.0203647832386196 1loss/SRqqt4/assignment_loss‚2.279642791748047 .loss/FRt1/detection_loss‚1.433952679622641 0loss/FRt2/assignment_loss‚2.3813408778773413 2loss/SRqqt2/assignment_loss‚1.9844903548558552 1loss/FBt1/assignment_loss‚0.10092910029457666 1loss/FBt2/assignment_loss‚0.20973170755960163 .loss/FRt4/detection_loss‚2.731965065905542 0loss/FRt1/assignment_loss‚2.3276133835315704 2loss/SRqqt1/assignment_loss‚1.2358798233426025 0loss/FBt3/assignment_loss‚0.5607547771424934 .loss/FRt2/detection_loss‚2.053872817920314 .loss/FRt3/detection_loss‚2.405578244955112 /loss/FBt3/detection_loss‚2.2612011631329856 trainer/global_step‚1849 epoch‚1 /loss/FRt4/assignment_loss‚4.194601449099454 1loss/SRqqt2/detection_loss‚2.4922801239622965 /loss/FBt2/detection_loss‚0.8096062652766705 &loss/total_loss‚46.103107268512105 1loss/SRqqt3/detection_loss‚1.6327812485396862 1loss/SRqqt4/detection_loss‚2.7728703367710112 .loss/FBt4/detection_loss‚2.348573638842656 ,loss/FBt1/detection_loss‚3.0525727125683 _step‚150 1loss/FBt4/assignment_loss‚0.16300282777788547 1loss/SRqqt1/detection_loss‚0.9856761389244826 _runtime‚3019.471172787 $ _timestamp‚1.788472812715808e+09£É~\ËÖ 1loss/SRqqt3/detection_loss‚1.6327812485396862 1loss/SRqqt4/detection_loss‚2.7728703367710112 .loss/FBt4/detection_loss‚2.348573638842656 ,loss/FBt1/detection_loss‚3.0525727125683 _step‚150 1loss/FBt4/assignment_loss‚0.16300282777788547 1loss/SRqqt1/detection_loss‚0.9856761389244826 _runtime‚3019.471172787 $ _timestamp‚1.788472812715808e+09 /loss/FRt3/assignment_loss‚3.663824955622355 2loss/SRqqt3/assignment_loss‚2.0203647832386196 1loss/SRqqt4/assignment_loss‚2.279642791748047 .loss/FRt1/detection_loss‚1.433952679622641 0loss/FRt2/assignment_loss‚2.3813408778773413 2loss/SRqqt2/assignment_loss‚1.9844903548558552 1loss/FBt1/assignment_loss‚0.10092910029457666 1loss/FBt2/assignment_loss‚0.20973170755960163 .loss/FRt4/detection_loss‚2.731965065905542 0loss/FRt1/assignment_loss‚2.3276133835315704 2loss/SRqqt1/assignment_loss‚1.2358798233426025 0loss/FBt3/assignment_loss‚0.5607547771424934 .loss/FRt2/detection_loss‚2.053872817920314 .loss/FRt3/detection_loss‚2.405578244955112 /loss/FBt3/detection_loss‚2.2612011631329856 trainer/global_step‚1849 epoch‚1 /loss/FRt4/assignment_loss‚4.194601449099454 1loss/SRqqt2/detection_loss‚2.4922801239622965 /loss/FBt2/detection_loss‚0.8096062652766705 &loss/total_loss‚46.103107268512105–%$ûlÌ‚(:á ÷ÛçÔ…‚ïx disk.sda3.out‚ 95.234375* proc.memory.percent‚20.516014099121094 network.sent‚ 3.889762e+06% disk./.usageGB‚1155.6344947814941 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 95.234375 disk.sda.in‚0$ proc.memory.availableMB‚6.390625 proc.cpu.threads‚17! disk.nvme0n1.in‚ 8596.42578125& disk.nvme0n1.out‚71318.36083984375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 8596.42578125( disk.nvme0c0n1.out‚71318.36083984375 disk.sda.out‚ 95.234375$ memory_percent‚99.96099472045898 cpu‚24.833709703478505 network.recv‚966716) disk./.usagePercent‚77.55560058671193 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4296875 disk.dm-0.in‚ 8596.42578125# disk.dm-0.out‚71318.36083984375 disk.sda3.in‚0 proc.memory.rssMB‚ 3361.34375 disk.sda2.in‚0 disk.sda2.out‚0±@o)œÍ‚(:‘ ÷ÛçÔ¯ñèy gpu.0.gpu‚15.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚266.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.378800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Œs’΂(:‡ †ÜçԌ˫x gpu.0.gpu‚62.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚233.237' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.6474 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±¶V‹hÏ‚(:Ý †ÜçÔØÿ–{ disk.dm-1.out‚95.59375 disk.sda3.in‚0$ memory_percent‚99.96073246002197& proc.memory.availableMB‚ 6.43359375 network.recv‚972084) disk./.usagePercent‚77.55930885584539 disk.sda1.out‚0# disk.dm-0.out‚71735.78662109375 disk.sda3.out‚95.59375 disk.drbd1260.in‚0 cpu‚24.83542104512502 proc.cpu.threads‚17( disk.nvme0c0n1.out‚71735.78662109375 disk.dm-0.in‚ 8745.6640625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 8745.6640625& disk.nvme0n1.out‚71735.78662109375 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 network.sent‚ 3.999543e+06% disk./.usageGB‚1155.6897506713867 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚95.59375 disk.drbd1260.out‚ 0.43359375 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 8745.6640625;áUmЂ(:â •ÜçÔŒÀ¿x& disk.nvme0n1.out‚71955.31103515625 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 96.1015625 disk.drbd1260.in‚0$ memory_percent‚99.96321201324463 disk.sda2.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚71955.31103515625 disk.dm-0.in‚ 8745.6640625# disk.dm-0.out‚71955.31103515625 disk.sda.out‚ 96.1015625 disk.sda3.in‚0 disk.sda3.out‚ 96.1015625 network.recv‚976935) disk./.usagePercent‚77.54984220924426 disk.nvme0n1.in‚ 8745.6640625 disk.dm-1.in‚0& proc.memory.availableMB‚ 6.02734375 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 cpu‚24.83735615275713" disk.nvme0c0n1.in‚ 8745.6640625 disk.sda.in‚0 disk.drbd1260.out‚ 0.43359375 proc.cpu.threads‚17 network.sent‚ 4.00375e+06% disk./.usageGB‚1155.5486907958984 ,~Ñ‚(:… •ÜçÔŒé°y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.107' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0н¾¹‘Ò‚(:† ¤ÜçÔ¹£àx gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÇ2[nÓ‚(:ã ¤ÜçÔãÕy cpu‚24.839059351024968 proc.cpu.threads‚17 disk.dm-0.in‚ 8745.6640625 network.sent‚ 4.008608e+06% disk./.usageGB‚1155.7845993041992 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 8745.6640625 disk.dm-1.out‚ 96.49609375 disk.sda3.out‚ 96.49609375 disk.nvme0n1.in‚ 8745.6640625 disk.sda2.out‚0# disk.dm-0.out‚72322.11181640625$ memory_percent‚99.96345043182373 proc.memory.rssMB‚ 3361.34375( disk.nvme0c0n1.out‚72322.11181640625 network.recv‚980541& disk.nvme0n1.out‚72322.11181640625 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0) disk./.usagePercent‚77.56567422717674 disk.dm-1.in‚0 disk.sda.out‚ 96.49609375 disk.drbd1260.out‚0.4375& proc.memory.availableMB‚ 5.98828125* proc.memory.percent‚20.516014099121094¹ên4vÔ‚(:ƒ ³ÜçÔ´®šx gpu.0.gpu‚10.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0&áÊû.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.05' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.21 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éàkÕ‚(:à ³ÜçÔþƒ—y disk.dm-1.out‚ 96.8203125 proc.memory.rssMB‚ 3361.34375 proc.cpu.threads‚17 network.sent‚ 4.015023e+06& disk.nvme0n1.out‚72481.14697265625 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 8745.6640625( disk.nvme0c0n1.out‚72481.14697265625 disk.sda.out‚ 96.8203125 disk.sda3.out‚ 96.8203125 disk.drbd1260.in‚0 network.recv‚985210 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.4375$ memory_percent‚99.96349811553955& proc.memory.availableMB‚ 5.98046875* proc.memory.percent‚20.516014099121094) disk./.usagePercent‚77.56056918093645 disk.nvme0n1.in‚ 8745.6640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 8745.6640625# disk.dm-0.out‚72481.14697265625 disk.dm-1.in‚0 cpu‚24.841073370549232% disk./.usageGB‚1155.7085304260254;øU’Ö‚(:‡ ÂÜçÔΘx gpu.0.gpu‚55.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚204.851' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.9702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#âçcׂ(:Ø ÂÜçÔÓ—æz proc.cpu.threads‚17 network.recv‚988710) disk./.usagePercent‚77.57097998332387% disk./.usageGB‚1155.8636589050293 disk.dm-0.in‚ 8748.40625! disk.dm-0.out‚72834.787109375 disk.dm-1.out‚ 97.2109375 disk.sda.in‚0 disk.sda3.out‚ 97.2109375 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 8748.40625$ disk.nvme0n1.out‚72834.787109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚99.96378421783447& proc.memory.availableMB‚ 5.93359375 network.sent‚ 4.019952e+06 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.44140625 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 8748.40625 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 disk.sda.out‚ 97.2109375& disk.nvme0c0n1.out‚72834.787109375 cpu‚24.843197626232783FÈÓ2Ø- _wandbruntime‚3106  _runtime‚3106›fž»ÅÙ¿ % lr-AdamW/pg1‚0.000645405769073814 % lr-AdamW/pg2‚0.000645405769073814 trainer/global_step‚1899 % _timestamp‚1.7884728995993264e+09 _runtime‚3106.354818781 _step‚151ÕB— ÊÚÄ % lr-AdamW/pg1‚0.000645405769073814 % lr-AdamW/pg2‚0.000645405769073814 trainer/global_step‚1899 % _timestamp‚1.7884728995993264e+09 _runtime‚3106.354818781 _step‚151—2¿JL2Û- _wandbruntime‚3106  _runtime‚3106x*ú'ÜôO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39796575 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296584799408841 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47372534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303619100624896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291976282268272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87142771 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4616567330304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270698168657408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4616567330304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6364041658880 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299006170345984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277062337652736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78847289960043e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25108623 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4612956670464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39796581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303622737676288 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39796575 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303622737676288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17234624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3886256640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4612956670464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299006143954432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42342514 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303619100624896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299006170345984 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1189 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4608491712601 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6364117792256 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47372534 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 _step‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17233896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270698219860480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 523944960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968=ÉÔ]ÿ'ÝùO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 523944960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39796575 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296584799408841 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47372534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303619100624896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291976282268272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87142771 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4616567330304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270698168657408 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4616567330304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6364041658880 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299006170345984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277062337652736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78847289960043e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25108623 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4612956670464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39796581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303622737676288 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39796575 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303622737676288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17234624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3886256640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4612956670464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299006143954432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42342514 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303619100624896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299006170345984 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1189 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4608491712601 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6364117792256 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47372534 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 _step‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17233896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270698219860480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344˜~-2Þ- _wandbruntime‚3107  _runtime‚3107¨,«Õ$߀N KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47370763 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270837510249472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39817005 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42364780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 523944960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42364075 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303775288665600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39816999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25121428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47397127 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2058 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39816999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299159921513472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4615367152128 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3886256640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4615367152128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299159921513472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87214132 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6367364397056 _runtime‚3107.389778724 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17243352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47370763 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296733900862619 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47397127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚705 % _timestamp‚1.7884729006334507e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87214132 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4618977825280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚292126636937376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4610898016095 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532µxÇ 'à…N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299159921513472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87214132 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6367364397056 _runtime‚3107.389778724 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17243352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47370763 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296733900862619 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47397127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚705 % _timestamp‚1.7884729006334507e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87214132 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4618977825280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚292126636937376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4610898016095 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47370763 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270837510249472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39817005 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42364780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 523944960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42364075 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303775288665600 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39816999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25121428 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚1899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47397127 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2058 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39816999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299159921513472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4615367152128 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3886256640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4615367152128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712™]ã½H2á- _wandbruntime‚3107  _runtime‚3107…é§ÞWâÑ /loss/FRt1/assignment_loss‚2.547471735384557 /loss/FRt4/detection_loss‚2.7318239957094193 /loss/FRt3/detection_loss‚3.2146811343374706 0loss/SRqqt3/detection_loss‚2.434729648213233 2loss/SRqqt3/assignment_loss‚1.8451087734391611 2loss/FBt1/assignment_loss‚0.009059928357601166 0loss/FBt2/assignment_loss‚0.3392707174909063 1loss/FBt4/assignment_loss‚0.34825850653252627 /loss/FRt1/detection_loss‚1.5814431403920828 .loss/FBt3/detection_loss‚1.940283667582732 .loss/FRt2/detection_loss‚2.493196801115305 1loss/SRqqt1/detection_loss‚1.2721619245062754 /loss/SRqqt2/detection_loss‚2.44947766015927 epoch‚1 /loss/FRt3/assignment_loss‚3.724455191975548 1loss/SRqqt4/assignment_loss‚2.146551412343979 _runtime‚3107.391420179 % _timestamp‚1.7884729006359205e+09 0loss/FRt2/assignment_loss‚3.2254210527126608 0loss/FBt3/assignment_loss‚0.5848611998147905 1loss/SRqqt4/detection_loss‚3.1157434687018393 .loss/FBt1/detection_loss‚4.198660393555959 /loss/FBt4/detection_loss‚2.3206358502308526 %loss/total_loss‚51.58882973412831 _step‚154 /loss/FRt4/assignment_loss‚4.212527099408601 2loss/SRqqt1/assignment_loss‚1.2652899474267243 1loss/SRqqt2/assignment_loss‚2.523113302886486 /loss/FBt2/detection_loss‚1.0646031818503294 trainer/global_step‚1899&#п\ãÖ /loss/FRt1/assignment_loss‚2.547471735384557 /loss/FRt4/detection_loss‚2.7318239957094193 /loss/FRt3/detection_loss‚3.2146811343374706 0loss/SRqqt3/detection_loss‚2.434729648213233 2loss/SRqqt3/assignment_loss‚1.8451087734391611 2loss/FBt1/assignment_loss‚0.009059928357601166 0loss/FBt2/assignment_loss‚0.3392707174909063 1loss/FBt4/assignment_loss‚0.34825850653252627 /loss/FRt1/detection_loss‚1.5814431403920828 .loss/FBt3/detection_loss‚1.940283667582732 .loss/FRt2/detection_loss‚2.493196801115305 1loss/SRqqt1/detection_loss‚1.2721619245062754 /loss/SRqqt2/detection_loss‚2.44947766015927 epoch‚1 /loss/FRt3/assignment_loss‚3.724455191975548 1loss/SRqqt4/assignment_loss‚2.146551412343979 _runtime‚3107.391420179 % _timestamp‚1.7884729006359205e+09 0loss/FRt2/assignment_loss‚3.2254210527126608 0loss/FBt3/assignment_loss‚0.5848611998147905 1loss/SRqqt4/detection_loss‚3.1157434687018393 .loss/FBt1/detection_loss‚4.198660393555959 /loss/FBt4/detection_loss‚2.3206358502308526 %loss/total_loss‚51.58882973412831 _step‚154 /loss/FRt4/assignment_loss‚4.212527099408601 2loss/SRqqt1/assignment_loss‚1.2652899474267243 1loss/SRqqt2/assignment_loss‚2.523113302886486 /loss/FBt2/detection_loss‚1.0646031818503294 trainer/global_step‚1899š¡r;Þdä‚(:Ù ÑÜçÔÛ—ê{ network.recv‚993287 disk.sda.out‚97.65625 disk.sda2.out‚0 disk.sda1.out‚0 proc.memory.availableMB‚5.75* proc.memory.percent‚20.516014099121094# disk.nvme0c0n1.in‚ 8858.73828125( disk.nvme0c0n1.out‚73182.59716796875 disk.dm-0.in‚ 8858.73828125 disk.dm-1.in‚0 disk.sda3.out‚97.65625! disk.nvme0n1.in‚ 8858.73828125& disk.nvme0n1.out‚73182.59716796875$ memory_percent‚99.96490478515625 network.sent‚ 4.025946e+06) disk./.usagePercent‚77.57517031464431$ disk./.usageGB‚1155.926097869873# disk.dm-0.out‚73182.59716796875 disk.dm-1.out‚97.65625 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.44140625 disk.sda2.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 3361.34375 cpu‚24.84515648420455 proc.cpu.threads‚17 disk.sda3.in‚0êÇ=¼Œå‚(: ÑÜçÔžÁ¿| gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.4' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.68 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©WoKpæ‚(:å àÜçÔ›£÷x% disk.nvme0n1.out‚73328.6728515625 disk.sda2.out‚0 disk.dm-1.out‚ 97.98828125 disk.sda.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 5.71484375 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 8858.73828125 disk.sda1.out‚0 disk.sda3.out‚ 97.98828125$ memory_percent‚99.96511936187744 proc.memory.rssMB‚ 3361.34375 cpu‚24.84715952619734 disk.dm-0.in‚ 8858.73828125 disk.dm-1.in‚0 disk.sda.out‚ 97.98828125* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 8858.73828125' disk.nvme0c0n1.out‚73328.6728515625 network.sent‚ 4.133834e+06% disk./.usageGB‚1155.9654159545898 disk.drbd1260.out‚ 0.4453125 disk.sda2.in‚0 disk.sda1.in‚0" disk.dm-0.out‚73328.6728515625 network.recv‚998941) disk./.usagePercent‚77.57780898429974îF‡\‘ç‚(:† àÜçÔáþˆy gpu.0.gpu‚10.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!8,5uè‚(:ê ïÜçÔ”x* proc.memory.percent‚20.516014099121094 network.recv‚ 1.004011e+06% disk.nvme0n1.out‚73844.5947265625 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 8858.73828125 disk.dm-1.out‚ 98.33984375 disk.sda.in‚0 disk.sda.out‚ 98.33984375( disk./.usagePercent‚77.5877886699995% disk./.usageGB‚1156.1141204833984 disk.sda2.in‚0" disk.dm-0.out‚73844.5947265625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4453125$ memory_percent‚99.96676445007324 disk.sda2.out‚0 disk.dm-0.in‚ 8858.73828125 disk.sda3.out‚ 98.33984375 proc.memory.rssMB‚ 3361.34375 cpu‚24.849188211511002 proc.cpu.threads‚17 network.sent‚ 4.140401e+06! disk.nvme0n1.in‚ 8858.73828125' disk.nvme0c0n1.out‚73844.5947265625 disk.sda3.in‚0% proc.memory.availableMB‚ 5.4453125ËÀÚZ’é‚(:‡ ïÜçÔäÌèx gpu.0.gpu‚88.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚116.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒNœyê‚(:î þÜçÔˆ÷¹x disk.dm-1.out‚ 98.75390625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.44921875* proc.memory.percent‚20.516014099121094) disk./.usagePercent‚77.58828455659331 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.out‚ 98.75390625 disk.sda3.out‚ 98.75390625 disk.drbd1260.in‚0& disk.nvme0n1.out‚74031.29345703125 network.sent‚ 4.145372e+06# disk.nvme0c0n1.in‚ 8858.73828125 disk.dm-0.in‚ 8858.73828125$ memory_percent‚99.96554851531982& proc.memory.availableMB‚ 5.64453125 proc.memory.rssMB‚ 3361.34375 cpu‚24.851103132880066 network.recv‚ 1.00727e+06$ disk./.usageGB‚1156.121509552002 disk.sda1.out‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 8858.73828125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚74031.29345703125# disk.dm-0.out‚74031.29345703125 disk.dm-1.in‚0ûm÷˜’ë‚(:‡ þÜçԈƶx gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚155.473' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.0946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î'^^pì‚(:å ÝçÔÂä¨x disk.dm-0.in‚ 8984.4921875) disk./.usagePercent‚77.58828327655564 disk.drbd1260.out‚ 0.44921875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 99.171875 disk.sda3.out‚ 99.171875$ memory_percent‚99.96557235717773$ proc.memory.availableMB‚5.640625( disk.nvme0c0n1.out‚74186.65283203125& disk.nvme0n1.out‚74186.65283203125 disk.sda1.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 3361.34375 cpu‚24.853007750918533 network.sent‚ 4.151577e+06 disk.sda1.out‚0# disk.dm-0.out‚74186.65283203125 disk.dm-1.out‚ 99.171875* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 8984.4921875 network.recv‚ 1.012303e+06% disk./.usageGB‚1156.1214904785156 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 8984.4921875 disk.sda2.in‚0 disk.sda2.out‚0VÅ8í‘í‚(:† ÝçÔ†èy gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚190.214' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.0428 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë+‚î‚(:… œÝçÔ¯µéx gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ATgOtï‚(:é œÝçÔ˜µšy disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 6.37109375 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 cpu‚24.85495555719979) disk./.usagePercent‚77.60505842624427$ disk./.usageGB‚1156.371452331543( disk.nvme0c0n1.out‚74534.82470703125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 99.76953125 disk.sda3.in‚0 disk.drbd1260.in‚0 network.sent‚ 4.156452e+06 network.recv‚ 1.015974e+06# disk.dm-0.out‚74534.82470703125 disk.dm-1.out‚ 99.76953125 disk.sda3.out‚ 99.76953125 disk.drbd1260.out‚0.453125 disk.nvme0n1.in‚ 8984.4921875& disk.nvme0n1.out‚74534.82470703125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.96111392974854 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 8984.4921875 disk.dm-0.in‚ 8984.4921875*dd2ð- _wandbruntime‚3207  _runtime‚3207ì›pÅñ¿ _step‚155 % lr-AdamW/pg1‚0.000644441353046743 % lr-AdamW/pg2‚0.000644441353046743 trainer/global_step‚1949 % _timestamp‚1.7884730009512212e+09 _runtime‚3207.707994224ëêÏøÊòÄ % lr-AdamW/pg1‚0.000644441353046743 % lr-AdamW/pg2‚0.000644441353046743 trainer/global_step‚1949 % _timestamp‚1.7884730009512212e+09 _runtime‚3207.707994224 _step‚155›^tý.2ó- _wandbruntime‚3207  _runtime‚3207tºGÓü'ôöO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40817775 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43424601 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25748868 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277662688107008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40817781 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4728820468109 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚156 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306695021906432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚306695048297984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚731 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311432154125312 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4737105827328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43423865 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311428517073920 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3886256640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 523944960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277662739310080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚306695021906432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48604236 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1949 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284192640866304 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89395673 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6529901556224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304217956130505 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40817781 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48577898 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17674997 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299492769753040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299492744325072jÌ ÛˆõûO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚311428517073920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89422017 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48577898 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚284192513529856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6529825422848 P@DeviceStatsMonitor.on_train´_>0y%_batch_start/requested_bytes.all.peak‚ 28880505153 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4737105827328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43423865 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311428517073920 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3886256640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 523944960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277662739310080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚306695021906432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48604236 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1949 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚284192640866304 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89395673 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6529901556224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304217956130505 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40817781 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48577898 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17674997 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299492769753040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299492744325072 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40817775 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43424601 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25748868 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277662688107008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40817781 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4728820468109 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚156 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚306695021906432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚306695048297984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚731 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚311432154125312 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚306848799465472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89467034 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25761673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731226771603 $ _timestamp‚1.788473003565271e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚306848799465472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17684510 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40838205 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4735905649152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277802029699072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6533204314112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311588342179328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4739516322304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277801978496000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚311588342179328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40838199 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6533128194048 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89440664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284335106690048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43445473 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚306848825857024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚306848825857024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 523944960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48628829 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40838205 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 _runtime‚3210.326643722 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43446188 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48602465 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299643098994176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89440664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48628829 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89467034 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40838199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304370691674927ÿDö8 'ú„N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25761678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17683800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4739516322304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚306848799465472 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89467034 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25761673 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731226771603 $ _timestamp‚1.788473003565271e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚306848799465472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17684510 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40838205 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4735905649152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2060 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277802029699072 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6533204314112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311588342179328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4739516322304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277801978496000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚311588342179328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40838199 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6533128194048 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89440664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284335106690048 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43445473 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚306848825857024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚306848825857024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 523944960 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48628829 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40838205 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 _runtime‚3210.326643722 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43446188 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48602465 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299643098994176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89440664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48628829 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89467034 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40838199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304370691674927 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3886256640¯˜ûv2û- _wandbruntime‚3210  _runtime‚3210h@YüÓ 2loss/SRqqt2/assignment_loss‚2.1285904854536057 %loss/total_loss‚48.91409517967571 epoch‚1 _runtime‚3210.332707221 /loss/FRt3/assignment_loss‚4.419762845039368 0loss/FBt1/assignment_loss‚0.3893959659617394 /loss/FBt4/detection_loss‚2.6654672485131483 trainer/global_step‚1949 % _timestamp‚1.7884730035759642e+09 0loss/FBt3/assignment_loss‚0.3694506383035332 1loss/FBt4/assignment_loss‚0.43388371190513414 1loss/SRqqt1/detection_loss‚0.9112331880877416 .loss/FBt1/detection_loss‚3.089200980961323 /loss/FRt2/assignment_loss‚2.668619508402688 /loss/FRt1/detection_loss‚1.4160003277202007 _step‚158 1loss/SRqqt3/assignment_loss‚2.114635093323886 /loss/FRt3/detection_loss‚3.3027183914184572 .loss/FRt4/detection_loss‚2.561258193105459 1loss/SRqqt2/detection_loss‚2.6343975245952604 1loss/SRqqt3/detection_loss‚2.2171685442328455 /loss/FBt2/detection_loss‚0.7897727821643153 1loss/SRqqt4/assignment_loss‚2.173947307570227 1loss/FBt2/assignment_loss‚0.31407230823102616 /loss/FBt3/detection_loss‚2.0899633752803006 /loss/FRt1/assignment_loss‚2.326266632804388 /loss/FRt4/assignment_loss‚4.076425695419312 /loss/FRt2/detection_loss‚2.3371893882751467 1loss/SRqqt4/detection_loss‚2.2711540861376402 2loss/SRqqt1/assignment_loss‚1.21352095676896471¢ìÒrýØ /loss/FRt1/assignment_loss‚2.326266632804388 /loss/FRt4/assignment_loss‚4.076425695419312 /loss/FRt2/detection_loss‚2.3371893882751467 1loss/SRqqt4/detection_loss‚2.2711540861376402 2loss/SRqqt1/assignment_loss‚1.2135209567689647 2loss/SRqqt2/assignment_loss‚2.1285904854536057 %loss/total_loss‚48.91409517967571 epoch‚1 _runti”n£(ìme‚3210.332707221 /loss/FRt3/assignment_loss‚4.419762845039368 0loss/FBt1/assignment_loss‚0.3893959659617394 /loss/FBt4/detection_loss‚2.6654672485131483 trainer/global_step‚1949 % _timestamp‚1.7884730035759642e+09 0loss/FBt3/assignment_loss‚0.3694506383035332 1loss/FBt4/assignment_loss‚0.43388371190513414 1loss/SRqqt1/detection_loss‚0.9112331880877416 .loss/FBt1/detection_loss‚3.089200980961323 /loss/FRt2/assignment_loss‚2.668619508402688 /loss/FRt1/detection_loss‚1.4160003277202007 _step‚158 1loss/SRqqt3/assignment_loss‚2.114635093323886 /loss/FRt3/detection_loss‚3.3027183914184572 .loss/FRt4/detection_loss‚2.561258193105459 1loss/SRqqt2/detection_loss‚2.6343975245952604 1loss/SRqqt3/detection_loss‚2.2171685442328455 /loss/FBt2/detection_loss‚0.7897727821643153 1loss/SRqqt4/assignment_loss‚2.173947307570227 1loss/FBt2/assignment_loss‚0.31407230823102616 /loss/FBt3/detection_loss‚2.0899633752803006žÊp3¶œþ‚(:‘ ºÝçÔúĽy gpu.0.gpu‚74.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚249.574' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚49.91480000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@9Úëwÿ‚(:ì ºÝçÔ·ÁÔz disk.dm-1.out‚ 100.7265625 proc.memory.rssMB‚ 3361.34375 cpu‚24.858802514901324 proc.cpu.threads‚17 network.sent‚ 4.272156e+06 network.recv‚ 1.026673e+06 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 100.7265625 disk.sda3.out‚ 100.7265625 disk.nvme0n1.in‚ 8984.4921875* proc.memory.percent‚20.516014099121094 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.45703125& disk.nvme0n1.out‚75122.27880859375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 8984.4921875( disk.nvme0c0n1.out‚75122.27880859375) disk./.usagePercent‚77.61686882182138 disk.dm-0.in‚ 8984.4921875 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚99.96449947357178& proc.memory.availableMB‚ 5.81640625$ disk./.usageGB‚1156.547435760498# disk.dm-0.out‚75122.27880859375ü®÷¢u€‚(:ê ÉÝçÔ »µy disk.sda.out‚ 101.140625 disk.sda3.out‚ 101.140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.45703125$ memory_percent‚99.96304512023926 cpu‚24.860805402658475 network.sent‚ 4.357801e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚75564.97021484375 disk.dm-0.in‚ 9119.19140625% proc.memory.availableMB‚ 6.0546875 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 network.recv‚ 1.03389e+06) disk./.usagePercent‚77.61840793911723$ disk./.usageGB‚1156.570369720459! disk.nvme0n1.in‚ 9119.19140625& disk.nvme0n1.out‚75564.97021484375 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 9119.19140625# disk.dm-0.out‚75564.97021484375 disk.dm-1.in‚0 disk.dm-1.out‚ 101.140625±:A,‘‚(:† ÉÝçÔˆë‰z gpu.0.gpu‚47.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØYÈðg‚‚(:Ü ØÝçÔ»Ñëw disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0$ memory_percent‚99.96299743652344! disk.nvme0n1.in‚ 9119.19140625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 9119.19140625 disk.sda.out‚101.625 disk.sda3.out‚101.625 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 9119.19140625" disk.dm-0.out‚75862.7919921875 disk.dm-1.out‚101.625 disk.drbd1260.out‚ 0.4609375* proc.memory.percent‚20.516014099121094 cpu‚24.862571679721892 proc.cpu.threads‚17% disk./.usageGB‚1156.8203887939453' disk.nvme0c0n1.out‚75862.7919921875 disk.sda.in‚0" proc.memory.availableMB‚6.0625 proc.memory.rssMB‚ 3361.34375 network.sent‚ 4.364548e+06 network.recv‚ 1.037151e+06) disk./.usagePercent‚77.63518692891887% disk.nvme0n1.out‚75862.7919921875É­cr‘ƒ‚(:† ØÝçÔ·™Åy gpu.0.gpu‚34.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚116.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òÙêqx„‚(:í çÝçÔÝû´y disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚76055.2568359375# memory_percent‚99.9645471572876) disk./.usagePercent‚77.66730153804411% disk./.usageGB‚1157.2989196777344 disk.dm-1.in‚0 disk.sda.out‚ 102.18359375 disk.sda3.out‚ 102.18359375 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 9119.19140625 disk.dm-0.in‚ 9119.19140625 network.sent‚ 4.369026e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.4609375' disk.nvme0c0n1.out‚76055.2568359375& proc.memory.availableMB‚ 5.80859375 cpu‚24.86412592410951 proc.cpu.threads‚17 network.recv‚ 1.041871e+06# disk.nvme0c0n1.in‚ 9119.19140625 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 disk.dm-1.out‚ 102.18359375% disk.nvme0n1.out‚76055.2568359375þá~X…‚(:… çÝçÔÓÞ³z gpu.0.gpu‚24.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.203 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SYÆp‘†‚(:† öÝçÔ¾ŠËx gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÂÅÚt‡‚(:é öÝçÔé€z network.recv‚ 1.044534e+06 disk.sda1.out‚0& disk.nvme0c0n1.out‚76306.064453125 disk.dm-0.in‚ 9119.19140625! disk.dm-0.out‚76306.064453125 disk.dm-1.in‚0 disk.sda.out‚ 102.4765625 disk.drbd1260.in‚0 network.sent‚ 4.373076e+06) disk./.usagePercent‚77.66716611005849% disk./.usageGB‚1157.2969017028809 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚99.96304512023926 proc.memory.rssMB‚ 3361.34375! disk.nvme0n1.in‚ 9119.19140625$ disk.nvme0n1.out‚76306.064453125 disk.sda1.in‚0 disk.dm-1.out‚ 102.4765625 disk.sda3.out‚ 102.4765625 disk.drbd1260.out‚ 0.46484375% proc.memory.availableMB‚ 6.0546875 cpu‚24.865087460937488 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 9119.19140625* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17@bÜuˆ‚(:ê …ÞçÔ‘•y disk.dm-1.out‚ 102.91796875 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚99.96306896209717 cpu‚24.866065703897522) disk./.usagePercent‚77.70085695757206 disk.sda3.out‚ 102.91796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17 network.sent‚ 4.379186e+06 disk.dm-0.in‚ 9238.7578125 disk.dm-1.in‚0$ disk.nvme0n1.out‚76748.720703125% disk./.usageGB‚1157.7989196777344& disk.nvme0c0n1.out‚76748.720703125 disk.sda.in‚0 disk.sda.out‚ 102.91796875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.46484375 disk.nvme0n1.in‚ 9238.7578125& proc.memory.availableMB‚ 6.05078125 network.recv‚ 1.049084e+06" disk.nvme0c0n1.in‚ 9238.7578125! disk.dm-0.out‚76748.7207031250ã1|‘‰‚(:† …ÞçÔÛ‰¡y gpu.0.gpu‚31.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚189.618' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.9236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ÍÄV2Š- _wandbruntime‚3307  _runtime‚3307ÿŸ® Ç‹Á & lr-AdamW/pg1‚0.0006434105589078439 & lr-AdamW/pg2‚0.0006434105589078439 trainer/global_step‚1999 % _timestamp‚1.7884731003415062e+09 _runtime‚3307.098099045 _step‚159†°-°ÌŒÆ _step‚159 & lr-AdamW/pg1‚0.0006434105589078439 & lr-AdamW/pg2‚0.0006434105589078439 trainer/global_step‚1999 % _timestamp‚1.7884731003415062e+09 _runtime‚3307.098099045Ÿr ®L2- _wandbruntime‚3307  _runtime‚3307«Üýü'ŽöO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4857644324352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49809600 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3892548096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41838975 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2060 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6695481288192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291322740047872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311854746942813 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314383899858432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6695403057664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291322610614272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314383926249984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41838981 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4854033664512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44505582 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319237933522944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18115740 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚307009206381872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚319241570574336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91674919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44504858 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 129433600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78230528àO'(ûO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311854746942813 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314383899858432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6695403057664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291322610614272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314383926249984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41838981 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4854033664512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44505582 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319237933522944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18115740 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚307009206381872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚319241570574336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91674919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44504858 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 129433600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78230528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4857644324352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49809600 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3892548096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41838975 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2060 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6695481288192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1191 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291322740047872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 íBT2- _wandbruntime‚3309  _runtime‚3309 –rŸ7 ‘€N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4847946864227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18125196 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4860054819328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314537677417472 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚319397758628352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4860054819328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91719936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26401928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291465323934208 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚161 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314537677417472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49834167 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26401923 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18124485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78217216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44526408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2060 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3309.170267256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41859399 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307159535623008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319397758628352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41859399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49860531 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314537703809024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 129420288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284766497945600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312011116578087 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319394121563648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284766549148672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41859405 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49860531 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚716 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4856444146176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91693566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91693566 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6698774785536 % _timestamp‚1.7884731024094973e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91719936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4847946864227 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18125196 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4860054819328 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚314537677417472 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚319397758628352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4860054819328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91719936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26401928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚291465323934208 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚161 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314537677417472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49834167 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26401923 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18124485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78217216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44526408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2060 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3309.170267256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41859399 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307159535623008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319397758628352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41859399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49860531 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314537703809024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 129420288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284766497945600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312011116578087 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319394121563648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284766549148672 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41859405 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49860531 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚716 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1191 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4856444146176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91693566 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024¡™5„2“- _wandbruntime‚3309  _runtime‚3309]RbCW”Ñ % _timestamp‚1.7884731024181888e+09 /loss/FRt1/assignment_loss‚2.472601924211748 2loss/SRqqt1/assignment_loss‚1.3860155292704963 1loss/SRqqt3/detection_loss‚2.3733397251175297 0loss/SRqqt4/detection_loss‚3.121306149583114 /loss/FRt1/detection_loss‚1.8067024723296203 .loss/FRt4/detection_loss‚2.663987468450497 /loss/FBt4/detection_loss‚2.7134928922904167 0loss/FRt2/assignment_loss‚3.2321995936688923 1loss/SRqqt2/assignment_loss‚1.730911709368229 2loss/SRqqt3/assignment_loss‚1.8599424350165552 _step‚162 1loss/FBt4/assignment_loss‚0.37817793143381057 epoch‚1 trainer/global_step‚1999 1loss/FBt3/assignment_loss‚0.36783505935112343 /loss/FRt3/assignment_loss‚4.816675859948863 /loss/FRt4/assignment_loss‚4.109711958811833 0loss/SRqqt2/detection_loss‚4.642608404159546 .loss/FBt2/detection_loss‚0.858495212653104 _runtime‚3309.175122468 2loss/SRqqt4/assignment_loss‚2.6442993816576505 1loss/FBt2/assignment_loss‚0.35001754378197747 .loss/FRt2/detection_loss‚2.156839461198875 .loss/FBt1/detection_loss‚4.229231953620911 %loss/total_loss‚55.17759948131913 0loss/FBt1/assignment_loss‚0.3556290853690977 /loss/FRt3/detection_loss‚3.1990718323251475 1loss/SRqqt1/detection_loss‚1.1473407857936986 .loss/FBt3/detection_loss‚2.561165111906388"Â2ˆ\•Ö % _timestamp‚1.7884731024181888e+09 /loss/FRt1/assignment_loss‚2.472601924211748 2loss/SRqqt1/assignment_loss‚1.3860155292704963 1loss/SRqqt3/detection_loss‚2.3733397251175297 0loss/SRqqt4/detection_loss‚3.121306149583114 /loss/FRt1/detection_loss‚1.8067024723296203 .loss/FRt4/detection_loss‚2.663987468450497 /loss/FBt4/detection_loss‚2.7134928922904167 0loss/FRt2/assignment_loss‚3.2321995936688923 1loss/SRqqt2/assignment_loss‚1.730911709368229 2loss/SRqqt3/assignment_loss‚1.8599424350165552 _step‚162 1loss/FBt4/assignment_loss‚0.37817793143381057 epoch‚1 trainer/global_step‚1999 1loss/FBt3/assignment_loss‚0.36783505935112343 /loss/FRt3/assignment_loss‚4.816675859948863 /loss/FRt4/assignment_loss‚4.109711958811833 0loss/SRqqt2/detection_loss‚4.642608404159546 .loss/FBt2/detection_loss‚0.858495212653104 _runtime‚3309.175122468 2loss/SRqqt4/assignment_loss‚2.6442993816576505 1loss/FBt2/assignment_loss‚0.35001754378197747 .loss/FRt2/detection_loss‚2.156839461198875 .loss/FBt1/detection_loss‚4.229231953620911 %loss/total_loss‚55.17759948131913 0loss/FBt1/assignment_loss‚0.3556290853690977 /loss/FRt3/detection_loss‚3.1990718323251475 1loss/SRqqt1/detection_loss‚1.1473407857936986 .loss/FBt3/detection_loss‚2.561165111906388¢üˆÄÔn–‚(:ã ”ÞçÔ¸ŠÞx disk.sda3.out‚ 103.640625 network.sent‚ 4.384055e+06 network.recv‚ 1.05257e+06) disk./.usagePercent‚77.70073381794809' disk.nvme0c0n1.out‚77262.7626953125 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚99.96330738067627% disk./.usageGB‚1157.7970848083496 disk.dm-0.in‚ 9238.7578125 disk.dm-1.out‚ 103.640625 disk.sda3.in‚0 disk.drbd1260.out‚0.46875 disk.sda1.in‚0& proc.memory.availableMB‚ 6.01171875 cpu‚24.866881001038223" disk.nvme0c0n1.in‚ 9238.7578125" disk.dm-0.out‚77262.7626953125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 9238.7578125 disk.sda2.in‚0 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 disk.sda.out‚ 103.640625% disk.nvme0n1.out‚77262.7626953125 disk.sda2.out‚0 proc.cpu.threads‚17E2xK—‚(:… ”ÞçÔ긬y gpu.0.gpu‚56.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚201.39' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.278 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§?Ãt˜‚(:é £ÞçÔõ¡¤y disk.dm-0.in‚ 9238.7578125" disk.dm-0.out‚77551.3681640625 disk.dm-1.out‚ 104.01171875$ memory_percent‚99.96333122253418% proc.memory.availableMB‚ 6.0078125 proc.memory.rssMB‚ 3361.34375 network.recv‚ 1.059717e+06% disk./.usageGB‚1157.8205528259277 disk.sda.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 9238.7578125* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.out‚0.46875 disk.nvme0n1.in‚ 9238.7578125 disk.sda1.out‚0 cpu‚24.867893451585566 network.sent‚ 4.493348e+06) disk./.usagePercent‚77.70230877629874 disk.sda.out‚ 104.01171875 disk.sda3.out‚ 104.01171875% disk.nvme0n1.out‚77551.3681640625 disk.sda2.in‚0' disk.nvme0c0n1.out‚77551.3681640625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0NðHŸ’™‚(:‡ £ÞçÔèøúy gpu.0.gpu‚61.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚207.581' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ•2|š‚(:… ²ÞçÔ…Ÿ£y gpu.0.gpu‚20.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚207.825' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DQús›‚(:è ²ÞçÔš…Šz disk.sda.out‚ 104.421875& disk.nvme0n1.out‚77609.69189453125 proc.cpu.threads‚17 network.recv‚ 1.062909e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.47265625$ memory_percent‚99.96333122253418 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 network.sent‚ 4.49827e+06% disk./.usageGB‚1157.8205528259277" disk.nvme0c0n1.in‚ 9238.7578125 disk.sda.in‚0 disk.sda3.out‚ 104.421875 disk.nvme0n1.in‚ 9238.7578125 disk.sda1.in‚0( disk.nvme0c0n1.out‚77609.69189453125 disk.dm-0.in‚ 9238.7578125 disk.dm-1.out‚ 104.421875% proc.memory.availableMB‚ 6.0078125 cpu‚24.868675410899442) disk./.usagePercent‚77.70230877629874 disk.sda2.in‚0 disk.sda1.out‚0# disk.dm-0.out‚77609.69189453125 disk.dm-1.in‚0sϘ jœ‚(:ß ÁÞçÔ²ÿÜx disk.drbd1260.in‚0 disk.nvme0n1.in‚ 9392.515625& disk.nvme0n1.out‚77932.28564453125 network.sent‚ 4.504425e+06 network.recv‚ 1.067247e+06% disk./.usageGB‚1157.7968521118164 disk.sda.in‚0 disk.sda.out‚ 104.80859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1260.out‚ 0.47265625 disk.sda1.out‚0( disk.nvme0c0n1.out‚77932.28564453125 proc.memory.availableMB‚6* proc.memory.percent‚20.516014099121094 cpu‚24.86977193648191 disk.dm-0.in‚ 9392.515625 disk.sda3.out‚ 104.80859375! disk.nvme0c0n1.in‚ 9392.515625! memory_percent‚99.96337890625 proc.memory.rssMB‚ 3361.34375 proc.cpu.threads‚17( disk./.usagePercent‚77.7007182014885# disk.dm-0.out‚77932.28564453125 disk.dm-1.in‚0 disk.dm-1.out‚ 104.80859375 disk.sda3.in‚0GB®¤›‚(: ÁÞçÔ¡êöy gpu.0.gpu‚45.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚209.66' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.931999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tUiž‚(:Þ ÐÞçÔûóÄx network.recv‚ 1.070599e+06 disk.dm-1.out‚ 105.58203125 disk.sda.in‚0 disk.drbd1260.out‚ 0.48046875 disk.nvme0n1.in‚ 9404.390625 disk.sda1.out‚0 proc.memory.availableMB‚6# disk./.usageGB‚1157.82320022583# disk.dm-0.out‚78377.43798828125! memory_percent‚99.96337890625 proc.memory.rssMB‚ 3361.34375 cpu‚24.870625517909065( disk./.usagePercent‚77.7024864455275 disk.sda.out‚ 105.58203125 disk.drbd1260.in‚0& disk.nvme0n1.out‚78377.43798828125 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 9404.390625* proc.memory.percent‚20.516014099121094 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 105.58203125 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 9404.390625( disk.nvme0c0n1.out‚78377.43798828125 proc.cpu.threads‚17 network.sent‚ 4.509248e+06† ‡HŸ‚(:„ ÐÞçÔƒ¹äx gpu.0.gpu‚33.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QI ‚(:… ßÞçÔ±÷ãx gpu.0.gpu‚53.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚203.21' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.642 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤Ñë"v¡‚(:ë ßÞçÔªØùy disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 9404.390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 105.94140625 disk.sda3.in‚0 disk.sda3.out‚ 105.94140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.48046875( disk.nvme0c0n1.out‚78667.10595703125# disk.dm-0.out‚78667.10595703125$ disk./.usageGB‚1157.823143005371 disk.sda.in‚0& disk.nvme0n1.out‚78667.10595703125 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 disk.nvme0n1.in‚ 9404.390625$ memory_percent‚99.96342658996582% proc.memory.availableMB‚ 5.9921875 cpu‚24.871519799442318 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 9404.390625 disk.dm-1.in‚0 disk.dm-1.out‚ 105.94140625 network.sent‚ 4.515565e+06 network.recv‚ 1.075405e+06) disk./.usagePercent‚77.70248260541447% «’¢‚(:‡ îÞçÔÑïÑw gpu.0.gpu‚75.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èªyv£‚(:ë îÞçÔŽÉÕx disk.drbd1260.out‚0.484375 disk.nvme0n1.in‚ 9404.390625& disk.nvme0n1.out‚79110.53173828125 disk.sda2.in‚0$ memory_percent‚99.96345043182373 network.sent‚ 4.520485e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚79110.53173828125# disk.dm-0.out‚79110.53173828125 disk.sda2.out‚0 proc.memory.rssMB‚ 3361.34375 cpu‚24.872374694883376 network.recv‚ 1.078924e+06% disk./.usageGB‚1157.8231620788574 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 5.98828125 proc.cpu.threads‚17) disk./.usagePercent‚77.70248388545215 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 9404.390625 disk.sda.out‚ 106.36328125 disk.drbd1260.in‚0* proc.memory.percent‚20.516014099121094 disk.dm-0.in‚ 9404.390625 disk.dm-1.out‚ 106.36328125 disk.sda3.out‚ 106.36328125ÑÓéú2¤- _wandbruntime‚3406  _runtime‚3406͵±ÓÆ¥À & lr-AdamW/pg1‚0.0006423136014730703 & lr-AdamW/pg2‚0.0006423136014730703 trainer/global_step‚2049 $ _timestamp‚1.788473199654763e+09 _runtime‚3406.410480505 _step‚163A,]E˦Š$ _timestamp‚1.788473199654763e+09 _runtime‚3406.410480505 _step‚163 & lr-AdamW/pg1‚0.0006423136014730703 & lr-AdamW/pg2‚0.0006423136014730703 trainer/global_step‚2049£¥¤p°2§- _wandbruntime‚3406  _runtime‚3406‡!}x’ ¨õO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51067640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322072804201984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1204 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51041302 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322072777810432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6860796282880 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 trainer/global_step‚2049 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42860175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚749 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42860181 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319491537755121 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327047349971968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42860181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298452650625536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51067640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298452523289088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6860872416256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314525693866640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45587990 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚322072777810432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27029373 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 _runtime‚ 3406.41196869 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322072804201984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4965869316449 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4969477979125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18559366 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327050987023360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 _step‚164 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314525668438672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 % _timestamp‚1.7884731996560705e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93901477 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2073 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291591727006208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42860175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45588739 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51041302 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291591778209280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327047349971968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4978182821376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51041302 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚322072777810432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6860796282880 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 trainer/global_step‚2049 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42860175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚749 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42860181 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319491537755121 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327047349971968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42860181 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚298452650625536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51067640 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298452523289088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6860872416256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314525693866640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45587990 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚322072777810432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27029373 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 _runtime‚ 3406.41196869 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322072804201984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4965869316449 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4969477979125 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18559366 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327050987023360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 _step‚164 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314525668438672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 % _timestamp‚1.7884731996560705e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93901477 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2073 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291591727006208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42860175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45588739 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51041302 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291591778209280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327047349971968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4978182821376 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93972838 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42880605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42880599 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327207175077376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18568182 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2049 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45609638 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚3408.717567903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚722 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291731068598272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42880599 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45610360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27042178 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6864106918400 % _timestamp‚1.7884732019559922e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51092233 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322226581761024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322226555369472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4976982643200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51065869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298595124313600 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93972838 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51092233 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93946468 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51065869 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4968275619735 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1204 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319644273299543 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93946468 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322226555369472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4980593316352 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42880605 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6864106918400 % _timestamp‚1.7884732019559922e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51092233 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚322226581761024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322226555369472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4976982643200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51065869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚298595124313600 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93972838 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51092233 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93946468 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51065869 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4968275619735 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1204 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319644273299543 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93946468 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322226555369472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4980593316352 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42880605 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93972838 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42880605 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42880599 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327207175077376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18568182 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2049 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45609638 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚3408.717567903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚722 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291731068598272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42880599 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45610360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27042178 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43881375 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322042155923440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52299342 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327131962658073 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 _runtime‚3509.186522744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚738 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚334856766420992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52273004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚2099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298556246455808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5089806734633 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298556297658880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96180723 % _timestamp‚1.7884733024297109e+09 _step‚168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚329761682153984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305582803487744 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327128328567429—µ—W¶ÁûO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27669618 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7026505828864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329761682153984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5086198071957 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5095110658560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚743 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19002784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5095110658560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27669623 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46673145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329761655762432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7026429695488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚334860403472384 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43881375 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚322042155923440 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52299342 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327131962658073 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 _runtime‚3509.186522744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚738 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚334856766420992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52273004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚2099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298556246455808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5089806734633 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298556297658880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes‰TeK .large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96180723 % _timestamp‚1.7884733024297109e+09 _step‚168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚329761682153984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚305582803487744 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327128328567429 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43881381 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322042130495472 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚334860403472384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5098721318400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43881381 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840¨=N-uœÂ‚(:‘ ×ßçÔ„·¿y gpu.0.gpu‚35.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.415399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-c"xÂ(:í ×ßçÔ§¡äy disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚99.96473789215088& proc.memory.availableMB‚ 5.77734375* proc.memory.percent‚20.516014099121094 disk.sda.in‚0 network.sent‚ 4.667493e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0$ disk.nvme0n1.out‚81258.130859375) disk./.usagePercent‚77.63294174284358 disk.dm-0.in‚ 9668.98046875 disk.sda.out‚ 109.33203125 disk.drbd1260.out‚ 0.49609375 proc.memory.rssMB‚ 3361.34375 cpu‚24.878085410683834 proc.cpu.threads‚17 network.recv‚ 1.117353e+06 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 9668.98046875! disk.dm-0.out‚81258.130859375 disk.dm-1.out‚ 109.33203125 disk.sda3.in‚0 disk.sda3.out‚ 109.33203125% disk./.usageGB‚1156.7869338989258& disk.nvme0c0n1.out‚81258.130859375! disk.nvme0n1.in‚ 9668.98046875E™€ð2Ä- _wandbruntime‚3511  _runtime‚3511á$‰'ÅÿM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27682428 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96199370 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335016591526400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43901805 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27682423 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305725256636416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327284698202703 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329915433321472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52323935 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1204 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚322192485164576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _step‚169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46693911 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5088604375243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78217216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5092213038127 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19012238 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322192459736608 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 trainer/global_step‚2099 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96225740 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚750 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5101131813376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5097521140224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327281064111851 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335016591526400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52323935 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335012954461696 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5092213038127 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19012238 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322192459736608 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 trainer/global_step‚2099 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96225740 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚750 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5101131813376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5097521140224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327281064111851 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335016591526400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52323935 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335012954461696 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329915433321472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52323935 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1204 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚322192485164576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _step‚169 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46693911 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5088604375243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78217216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0©1îº2Ç- _wandbruntime‚3511  _runtime‚3511a¶ìÊUÈÏ 1loss/SRqqt3/assignment_loss‚2.989212379736059 -loss/FRt3/detection_loss‚3.76433545701644 0loss/SRqqt3/detection_loss‚2.641384952208575 .loss/FBt1/detection_loss‚5.889669418334961 /loss/FRt2/assignment_loss‚3.996231970461932 0loss/FBt2/assignment_loss‚0.2508038455052755 0loss/FBt3/assignment_loss‚0.4062045340172269 % _timestamp‚1.7884733044931116e+09 _step‚170 1loss/FBt1/assignment_loss‚0.15612317342311144 /loss/FBt3/detection_loss‚2.3830491710792887 /loss/FRt4/detection_loss‚2.5291080989089667 0loss/SRqqt4/detection_loss‚2.483238593146608 &loss/total_loss‚62.512702892487134 /loss/FRt1/assignment_loss‚3.144581963334765 2loss/SRqqt4/assignment_loss‚2.5827260178488656 /loss/FBt2/detection_loss‚0.9174329917615568 .loss/FBt4/detection_loss‚2.424836305777232 1loss/FBt4/assignment_loss‚0.38197889723375433 .loss/FRt1/detection_loss‚1.817983855937408 /loss/FRt3/assignment_loss‚7.115365869858685 2loss/SRqqt1/assignment_loss‚1.8611889941394328 .loss/FRt2/detection_loss‚2.630108429626985 0loss/FRt4/assignment_loss‚4.7230259717679495 0loss/SRqqt1/detection_loss‚1.104780241370201 1loss/SRqqt2/detection_loss‚3.0387217324713003 epoch‚1 trainer/global_step‚2099 _runtime‚3511.248717471 2loss/SRqqt2/assignment_loss‚3.2806100275205528’ߨZÉÔ 0loss/FRt4/assignment_loss‚4.7230259717679495 0loss/SRqqt1/detection_loss‚1.104780241370201 1loss/SRqqt2/detection_loss‚3.0387217324713003 epoch‚1 trainer/global_step‚2099 _runtime‚3511.248717471 2loss/SRqqt2/assignment_loss‚3.2806100275205528 1loss/SRqqt3/assignment_loss‚2.989212379736059 -loss/FRt3/detection_loss‚3.76433545701644 0loss/SRqqt3/detection_loss‚2.641384952208575 .loss/FBt1/detection_loss‚5.889669418334961 /loss/FRt2/assignment_loss‚3.996231970461932 0loss/FBt2/assignment_loss‚0.2508038455052755 0loss/FBt3/assignment_loss‚0.4062045340172269 % _timestamp‚1.7884733044931116e+09 _step‚170 1loss/FBt1/assignment_loss‚0.15612317342311144 /loss/FBt3/detection_loss‚2.3830491710792887 /loss/FRt4/detection_loss‚2.5291080989089667 0loss/SRqqt4/detection_loss‚2.483238593146608 &loss/total_loss‚62.512702892487134 /loss/FRt1/assignment_loss‚3.144581963334765 2loss/SRqqt4/assignment_loss‚2.5827260178488656 /loss/FBt2/detection_loss‚0.9174329917615568 .loss/FBt4/detection_loss‚2.424836305777232 1loss/FBt4/assignment_loss‚0.38197889723375433 .loss/FRt1/detection_loss‚1.817983855937408 /loss/FRt3/assignment_loss‚7.115365869858685 2loss/SRqqt1/assignment_loss‚1.8611889941394328 .loss/FRt2/detection_loss‚2.630108429626985ªw™zÊ‚(:ï æßçÔôøy# disk.nvme0c0n1.in‚ 9668.98046875 disk.sda.in‚0# memory_percent‚99.9654769897461) disk./.usagePercent‚77.63294174284358! disk.nvme0n1.in‚ 9668.98046875 disk.sda1.out‚0# disk.dm-0.out‚81316.64990234375 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17 network.sent‚ 4.763911e+06 network.recv‚ 1.122814e+06 disk.sda3.in‚0 disk.sda3.out‚ 109.73828125 disk.sda2.in‚0 disk.dm-0.in‚ 9668.98046875 disk.sda.out‚ 109.73828125# proc.memory.availableMB‚5.65625 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚81316.64990234375 disk.dm-1.in‚0 disk.dm-1.out‚ 109.73828125 cpu‚24.878806973082817 disk.sda2.out‚0 disk.sda1.in‚0% disk./.usageGB‚1156.7869338989258 disk.drbd1260.out‚ 0.50390625& disk.nvme0n1.out‚81316.64990234375×”óH›Ë‚(: æßçÔµ“ßy gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.323' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.464599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¸ýÝyÌ‚(:î õßçÔ¬àÃx) disk./.usagePercent‚77.63294174284358& disk.nvme0n1.out‚81566.05615234375 disk.sda1.out‚0 disk.dm-0.in‚ 9668.98046875 disk.dm-1.out‚ 110.1171875 disk.sda.in‚0 disk.drbd1260.out‚ 0.50390625" memory_percent‚99.965500831604 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 110.1171875 disk.sda3.out‚ 110.1171875 disk.drbd1260.in‚0 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 cpu‚24.879590492095783# disk.dm-0.out‚81566.05615234375 network.sent‚ 4.862478e+06 network.recv‚ 1.129495e+06# disk.nvme0c0n1.in‚ 9668.98046875( disk.nvme0c0n1.out‚81566.05615234375 proc.cpu.threads‚17% disk./.usageGB‚1156.7869338989258! disk.nvme0n1.in‚ 9668.98046875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 5.65234375¤8ÑçÍ‚(:… õßçÔöÐðx gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øñþ¨l΂(:á „àçÔÊ͸x network.recv‚ 1.132949e+06) disk./.usagePercent‚77.63290282969838 disk.dm-0.in‚ 9807.98828125 disk.sda.out‚ 110.546875 disk.sda1.in‚0& proc.memory.availableMB‚ 5.65234375 proc.cpu.threads‚17 network.sent‚ 4.867214e+06 disk.dm-1.out‚ 110.546875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0* proc.memory.percent‚20.516014099121094% disk./.usageGB‚1156.7863540649414 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd1260.out‚ 0.5078125! disk.nvme0n1.in‚ 9807.98828125 disk.sda2.in‚0 disk.sda1.out‚0" memory_percent‚99.965500831604# disk.nvme0c0n1.in‚ 9807.98828125% disk.nvme0c0n1.out‚81781.88671875 disk.dm-0.out‚81781.88671875 disk.sda3.out‚ 110.546875# disk.nvme0n1.out‚81781.88671875 proc.memory.rssMB‚ 3361.34375 cpu‚24.880428759080882Ï`ª³‘Ï‚(:† „àçÔ•Ó÷x gpu.0.gpu‚47.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚164.159' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8318 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ,ò¡lЂ(:á “àçÔçÐøw disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 82127.3984375 proc.memory.rssMB‚ 3361.34375 disk.dm-0.in‚ 9807.98828125 disk.dm-1.out‚ 111.1171875 disk.sda.in‚0! disk.nvme0n1.in‚ 9807.98828125" disk.nvme0n1.out‚ 82127.3984375 disk.sda1.in‚0* proc.memory.percent‚20.516014099121094 cpu‚24.88112973529085 network.recv‚ 1.138867e+06) disk./.usagePercent‚77.63290129365318% disk./.usageGB‚1156.7863311767578 disk.sda3.out‚ 111.1171875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.5078125$ memory_percent‚99.96542930603027 proc.cpu.threads‚17 disk.dm-0.out‚ 82127.3984375 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 9807.98828125% proc.memory.availableMB‚ 5.6640625 network.sent‚ 4.874893e+06 disk.sda.out‚ 111.1171875 disk.sda3.in‚0­î@Ñ‚(:… “àçÔ¿üµy gpu.0.gpu‚4.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMWÑ ¢emoryErrors‚0Þ¨÷ð‘Ò‚(:† ¢àçÔºô«w gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚267.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ÕlÞzÓ‚(:ï ¢àçÔÙïñx disk.sda3.out‚ 111.6796875$ memory_percent‚99.96562004089355 cpu‚24.881872311557686 network.recv‚ 1.141898e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.51171875 disk.sda1.out‚0( disk.nvme0c0n1.out‚82354.07861328125 disk.dm-0.in‚ 9807.98828125 disk.dm-1.in‚0% disk./.usageGB‚1156.3524742126465 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.dm-0.out‚82354.07861328125 disk.dm-1.out‚ 111.6796875% proc.memory.availableMB‚ 5.6328125 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 network.sent‚ 4.879606e+06) disk./.usagePercent‚77.60378478876142! disk.nvme0n1.in‚ 9807.98828125& disk.nvme0n1.out‚82354.07861328125 disk.sda2.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 9807.98828125 proc.cpu.threads‚17 disk.sda.out‚ 111.6796875»u œÔ‚(:‘ ±àçÔÇßïw gpu.0.gpu‚77.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.362' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.27240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ^0„~Õ‚(:ó ±àçÔljy disk.sda.out‚ 112.12890625 disk.drbd1260.out‚ 0.51171875) disk./.usagePercent‚77.60378120465595# disk.nvme0c0n1.in‚ 9807.98828125 disk.dm-1.out‚ 112.12890625 disk.sda.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 9807.98828125& disk.nvme0n1.out‚82747.71533203125$ memory_percent‚99.96569156646729 network.recv‚ 1.146346e+06% disk./.usageGB‚1156.3524208068848 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 112.12890625 disk.drbd1260.in‚0 cpu‚24.882687440650777 proc.cpu.threads‚17 network.sent‚ 4.885658e+06& proc.memory.availableMB‚ 5.62109375 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚82747.71533203125 disk.dm-0.in‚ 9807.98828125# disk.dm-0.out‚82747.71533203125 disk.dm-1.in‚0y„A}2Ö- _wandbruntime‚3607  _runtime‚3607/JæþÇ×Á & lr-AdamW/pg2‚0.0006399221248737313 trainer/global_step‚2149 % _timestamp‚1.7884734010521836e+09 _runtime‚3607.808831241 _step‚171 & lr-AdamW/pg1‚0.0006399221248737313=xi.ÌØÆ & lr-AdamW/pg2‚0.0006399221248737313 trainer/global_step‚2149 % _timestamp‚1.7884734010521836e+09 _runtime‚3607.808831241 _step‚171 & lr-AdamW/pg1‚0.0006399221248737313«œ(ÌÆ2Ù- _wandbruntime‚3607  _runtime‚3607Tq Úü'ÚöO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28309868 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44902575 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5215649155584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47756479 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28309873 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚305520765905408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚342669819921408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚337450560105984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚312712640667136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329558617980240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _runtime‚3607.813157797 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 % _timestamp‚1.7884734010558312e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5219259815424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342666182870016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7191745328128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5215649155584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47755750 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98433625 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19446606 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334768753470381 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337450533714432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44902581 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342666182870016 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305520817108480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚329558592552272 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚724 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5210135490141 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 _step‚172 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98433625 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19446606 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334768753470381 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337450533714432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44902581 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342666182870016 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305520817108480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚329558592552272 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚724 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5210135490141 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 _step‚172 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚329558617980240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _runtime‚3607.813157797 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 % _timestamp‚1.7884734010558312e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5219259815424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024¬ï0†à2Ü- _wandbruntime‚3610  _runtime‚3610š |ëK#ÝÿM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98478642 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334921489015011 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305660107497472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47778118 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7195052849152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚329708921793408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚173 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5221670310400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28322678 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53529273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312855109143552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚337604311273472 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98478642 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305660056294400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19454731 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 _runtime‚3610.212522503 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7195128969216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19455440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5218059637248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312855236466688 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 trainer/global_step‚2149 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5218059637248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44922999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98452272 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98452272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53555637 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5221670310400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47777404 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329708947221376 T@Dð5)ºeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚342826007975424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53529273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5208933130751 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53555637 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342822370910720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44922999¸Yì 'Þ„N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98478642 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334921489015011 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚305660107497472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47778118 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7195052849152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚329708921793408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚173 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5221670310400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28322678 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53529273 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312855109143552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚337604311273472 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98478642 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305660056294400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76120064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19454731 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 _runtime‚3610.212522503 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7195128969216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19455440 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5218059637248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312855236466688 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 trainer/global_step‚2149 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5218059637248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44922999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98452272 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98452272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53555637 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5221670310400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47777404 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329708947221376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚342826007975424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53529273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5208933130751 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53555637 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342822370910720 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44922999 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370­›Gª2ß- _wandbruntime‚3610  _runtime‚3610œ˜FSàÍ /loss/FRt1/assignment_loss‚2.349279965673174 /loss/FRt3/assignment_loss‚7.160994922413545 %loss/total_loss‚57.42375505680849 trainer/global_step‚2149 2loss/SRqqt4/assignment_loss‚2.3539084789653617 0loss/FBt3/assignment_loss‚0.5924652148710771 /loss/FRt1/detection_loss‚2.0202016334087602 /loss/FRt4/detection_loss‚2.7303922835149264 0loss/SRqqt1/detection_loss‚0.905349844833836 1loss/SRqqt2/assignment_loss‚2.161512076854706 /loss/FRt2/detection_loss‚2.9023000206798315 /loss/FRt4/assignment_loss‚4.285964416830163 1loss/FBt4/assignment_loss‚0.24484498593483117 _runtime‚3610.216149159 1loss/SRqqt2/detection_loss‚3.0148275196552277 /loss/FBt4/detection_loss‚2.5169947892427444 % _timestamp‚1.7884734034596703e+09 /loss/FRt2/assignment_loss‚3.189960427582264 0loss/SRqqt4/detection_loss‚2.834658602873484 epoch‚1 .loss/FBt3/detection_loss‚2.240718228949441 2loss/SRqqt1/assignment_loss‚1.7484842457342893 1loss/SRqqt3/assignment_loss‚2.277798896898394 0loss/FBt1/assignment_loss‚0.6156375778373331 0loss/FBt2/assignment_loss‚0.1721220864047162 .loss/FRt3/detection_loss‚3.714940968681784 0loss/SRqqt3/detection_loss‚2.932730070922686 _step‚174 .loss/FBt1/detection_loss‚3.261787007252375 .loss/FBt2/detection_loss‚1.195880790793542úWwXáÒ .loss/FBt1/detection_loss‚3.261787007252375 .loss/FBt2/detection_loss‚1.195880790793542 /loss/FRt1/assignment_loss‚2.349279965673174 /loss/FRt3/assignment_loss‚7.160994922413545 %loss/total_loss‚57.42375505680849 trainer/global_step‚2149 2loss/SRqqt4/assignment_loss‚2.3539084789653617 0loss/FBt3/assignment_loss‚0.5924652148710771 /loss/FRt1/detection_loss‚2.0202016334087602 /loss/FRt4/detection_loss‚2.7303922835149264 0loss/SRqqt1/detection_loss‚0.905349844833836 1loss/SRqqt2/assignment_loss‚2.161512076854706 /loss/FRt2/detection_loss‚2.9023000206798315 /loss/FRt4/assignment_loss‚4.285964416830163 1loss/FBt4/assignment_loss‚0.24484498593483117 _runtime‚3610.216149159 1loss/SRqqt2/detection_loss‚3.0148275196552277 /loss/FBt4/detection_loss‚2.5169947892427444 % _timestamp‚1.7884734034596703e+09 /loss/FRt2/assignment_loss‚3.189960427582264 0loss/SRqqt4/detection_loss‚2.834658602873484 epoch‚1 .loss/FBt3/detection_loss‚2.240718228949441 2loss/SRqqt1/assignment_loss‚1.7484842457342893 1loss/SRqqt3/assignment_loss‚2.277798896898394 0loss/FBt1/assignment_loss‚0.6156375778373331 0loss/FBt2/assignment_loss‚0.1721220864047162 .loss/FRt3/detection_loss‚3.714940968681784 0loss/SRqqt3/detection_loss‚2.932730070922686 _step‚174®9¦‹›â‚(: ÀàçÔÑЗx gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.323' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.464599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U Vzã‚(:ï ÀàçÔ٠Ȁ% disk./.usageGB‚1156.3525009155273# disk.dm-0.out‚82901.23974609375 disk.dm-1.out‚ 112.6015625 disk.sda3.in‚0& disk.nvme0n1.out‚82901.23974609375 disk.sda2.in‚0& proc.memory.availableMB‚ 5.33984375 cpu‚24.883461335669413 network.recv‚ 1.148878e+06( disk.nvme0c0n1.out‚82901.23974609375 disk.dm-1.in‚0 disk.sda3.out‚ 112.6015625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.515625 disk.sda2.out‚0 network.sent‚ 4.889645e+06# disk.nvme0c0n1.in‚ 9917.23046875 disk.sda1.out‚0 proc.memory.rssMB‚ 3361.34375 proc.cpu.threads‚17) disk./.usagePercent‚77.60378658081416 disk.dm-0.in‚ 9917.23046875 disk.sda.in‚0 disk.sda.out‚ 112.6015625! disk.nvme0n1.in‚ 9917.23046875 disk.sda1.in‚0$ memory_percent‚99.96740818023682* proc.memory.percent‚20.516014099121094 ¼ätä‚(:é ÏàçÔ“ÎÁx( disk.nvme0c0n1.out‚82927.31005859375# disk.dm-0.out‚82927.31005859375 disk.drbd1260.out‚0.515625$ memory_percent‚99.96719360351562! disk.nvme0n1.in‚ 9917.23046875 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚20.516014099121094 cpu‚24.884176812589043 network.recv‚ 1.176578e+06) disk./.usagePercent‚77.59096085935626 disk.dm-1.in‚0 disk.dm-1.out‚ 112.9921875 disk.sda.in‚0 proc.memory.rssMB‚ 3361.34375 proc.cpu.threads‚17% disk./.usageGB‚1156.1613883972168 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 9917.23046875 disk.dm-0.in‚ 9917.23046875 disk.sda.out‚ 112.9921875 disk.sda3.out‚ 112.9921875 disk.drbd1260.in‚0! proc.memory.availableMB‚5.375 network.sent‚ 5.009706e+06& disk.nvme0n1.out‚82927.31005859375 disk.sda2.out‚0 disk.sda1.in‚0É-è”å‚(:… Ïàçԃʀy gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùŸ qsæ‚(:è ÞàçÔ”¯äx disk.sda.out‚ 113.5078125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.51953125$ disk.nvme0n1.out‚83273.208984375& disk.nvme0c0n1.out‚83273.208984375 proc.memory.rssMB‚ 3361.34375 disk.dm-1.out‚ 113.5078125 disk.sda.in‚0 disk.sda3.out‚ 113.5078125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0! disk.dm-0.out‚83273.208984375% disk./.usageGB‚1156.1849327087402! disk.nvme0n1.in‚ 9917.23046875# disk.nvme0c0n1.in‚ 9917.23046875 disk.dm-0.in‚ 9917.23046875$ memory_percent‚99.96724128723145 network.recv‚ 1.186483e+06) disk./.usagePercent‚77.59254093785759 disk.sda1.in‚0% proc.memory.availableMB‚ 5.3671875* proc.memory.percent‚20.516014099121094 cpu‚24.884971410422924 proc.cpu.threads‚17 network.sent‚ 5.01801e+06 disk.dm-1.in‚0Å_I5‘ç‚(:† ÞàçÔé÷͉ gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5962 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÐQœè‚(:‘ íàçÔüíÞw gpu.0.gpu‚88.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.354' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.67079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QM jué‚(:ê íàçÔßäÄx network.sent‚ 5.025382e+06( disk./.usagePercent‚77.5992708639177# disk.nvme0c0n1.in‚ 9917.23046875& disk.nvme0c0n1.out‚83937.798828125 disk.dm-0.in‚ 9917.23046875 disk.sda.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 5.3671875 network.recv‚ 1.192563e+06! disk.dm-0.out‚83937.798828125 disk.sda3.in‚0 disk.sda3.out‚ 113.94921875 disk.drbd1260.out‚ 0.51953125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 113.94921875! disk.nvme0n1.in‚ 9917.23046875$ disk.nvme0n1.out‚83937.798828125$ disk./.usageGB‚1156.285213470459 disk.dm-1.out‚ 113.94921875 disk.drbd1260.in‚0$ memory_percent‚99.96724128723145 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 cpu‚24.885700206762035ÆATzê‚(:ï üàçÔî˜óy disk.drbd1260.in‚0% disk.nvme0n1.out‚84338.3291015625 disk.sda2.in‚0 disk.sda1.in‚0 network.sent‚ 5.030363e+06% disk./.usageGB‚1156.3606300354004" disk.dm-0.out‚84338.3291015625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 3361.34375 network.recv‚ 1.196046e+06 disk.dm-0.in‚ 10043.0546875! disk.nvme0n1.in‚ 10043.0546875 disk.sda1.out‚0$ memory_percent‚99.96721744537354 cpu‚24.886468020452156 proc.cpu.threads‚17) disk./.usagePercent‚77.60433213286963# disk.nvme0c0n1.in‚ 10043.0546875' disk.nvme0c0n1.out‚84338.3291015625 disk.drbd1260.out‚ 0.5234375& proc.memory.availableMB‚ 5.37109375* proc.memory.percent‚20.516014099121094 disk.dm-1.out‚ 114.30078125 disk.sda.out‚ 114.30078125 disk.sda3.in‚0 disk.sda3.out‚ 114.30078125DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350475599319040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350479236370432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48839181 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342401910192045 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5336187652608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45923781 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28950123 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45923775 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5336187652608 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54736408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚345139438057984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45923781 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319843152164864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7357815606784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342405544282689 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312485285355008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337075054609072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5339798312448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350479236370432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5339798312448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0QÀ(õÿO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319843152164864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7357815606784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚342405544282689 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312485285355008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚337075054609072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5339798312448 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350479236370432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5339798312448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28950118 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54736408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19889767 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350475599319040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350479236370432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48839181 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚342401910192045 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5336187652608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45923781 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28950123 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45923775 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5336187652608 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54736408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚345139438057984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45923781 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187°z¢„\2ö- _wandbruntime‚3712  _runtime‚3712\-§m '÷ƒN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19898607 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312624575744000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚345293215617024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54760975 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28962923 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5338598134272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45944199 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚716 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100731544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345293189225472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48861535 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48860814 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342558279827319 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 % _timestamp‚1.7884735056848633e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45944205 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45944205 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319985607669760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5332870549143 trainer/global_step‚2199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350635424424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19897891 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚350631787359744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350635424424448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100731544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5338598134272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚177 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚345293189225472 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5342208807424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7361031925760 _runtime‚ 3712.44173623 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚345293215617024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100705174 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28962928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45944199 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319985732895744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337225409278176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337225383850208 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54787339 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54787339 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7361105948672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚721 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48860814 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342558279827319 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 % _timestamp‚1.7884735056848633e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45944205 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45944205 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319985607669760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5332870549143 trainer/global_step‚2199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350635424424448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19897891 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚350631787359744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350635424424448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100731544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5338598134272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚177 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚345293189225472 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5342208807424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7361031925760 _runtime‚ 3712.44173623 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚345293215617024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100705174 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28962928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45944199 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319985732895744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337225409278176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337225383850208 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54787339 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54787339 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7361105948672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚721 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚716 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100731544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719±Ÿ ç2ù- _wandbruntime‚3712  _runtime‚3712š¼KRúÌ 0loss/FBt2/assignment_loss‚0.2892677990248915 0loss/FBt4/assignment_loss‚0.4251965235785714 /loss/FRt4/detection_loss‚2.5124116338168583 1loss/SRqqt1/detection_loss‚0.9973310195227973 /loss/SRqqt2/detection_loss‚2.63765532320196 0loss/SRqqt3/detection_loss‚2.500300041560469 0loss/SRqqt4/detection_loss‚2.538014054298401 &loss/total_loss‚54.691811099829756 % _timestamp‚1.7884735056882932e+09 /loss/FRt1/assignment_loss‚2.270748980343342 2loss/SRqqt1/assignment_loss‚1.4428343021780508 trainer/global_step‚2199 1loss/FBt1/assignment_loss‚0.19276713447955748 0loss/FBt3/assignment_loss‚0.5041823574568601 epoch‚1 _step‚178 .loss/FRt2/detection_loss‚2.041788377907387 /loss/FRt3/assignment_loss‚7.339111857944065 /loss/FRt1/detection_loss‚1.4892147127662612 .loss/FBt1/detection_loss‚4.224133312702179 /loss/FBt2/detection_loss‚0.9609081299806183 -loss/FBt3/detection_loss‚2.04573871816198 _runtime‚3712.443899025 0loss/FRt2/assignment_loss‚2.6947267499080922 /loss/FRt4/assignment_loss‚4.256015710532665 2loss/SRqqt2/assignment_loss‚1.9377773837609724 0loss/SRqqt4/assignment_loss‚2.05667866114527 .loss/FRt3/detection_loss‚4.529086887836456 .loss/FBt4/detection_loss‚2.723431336028235 2loss/SRqqt3/assignment_loss‚2.0824900916938125»Ö“á?ûÑ -loss/FBt3/detection_loss‚2.04573871816198 _runti¨1”†me‚3712.443899025 0loss/FRt2/assignment_loss‚2.6947267499080922 /loss/FRt4/assignment_loss‚4.256015710532665 2loss/SRqqt2/assignment_loss‚1.9377773837609724 0loss/SRqqt4/assignment_loss‚2.05667866114527 .loss/FRt3/detection_loss‚4.529086887836456 .loss/FBt4/detection_loss‚2.723431336028235 2loss/SRqqt3/assignment_loss‚2.0824900916938125 0loss/FBt2/assignment_loss‚0.2892677990248915 0loss/FBt4/assignment_loss‚0.4251965235785714 /loss/FRt4/detection_loss‚2.5124116338168583 1loss/SRqqt1/detection_loss‚0.9973310195227973 /loss/SRqqt2/detection_loss‚2.63765532320196 0loss/SRqqt3/detection_loss‚2.500300041560469 0loss/SRqqt4/detection_loss‚2.538014054298401 &loss/total_loss‚54.691811099829756 % _timestamp‚1.7884735056882932e+09 /loss/FRt1/assignment_loss‚2.270748980343342 2loss/SRqqt1/assignment_loss‚1.4428343021780508 trainer/global_step‚2199 1loss/FBt1/assignment_loss‚0.19276713447955748 0loss/FBt3/assignment_loss‚0.5041823574568601 epoch‚1 _step‚178 .loss/FRt2/detection_loss‚2.041788377907387 /loss/FRt3/assignment_loss‚7.339111857944065 /loss/FRt1/detection_loss‚1.4892147127662612 .loss/FBt1/detection_loss‚4.224133312702179 /loss/FBt2/detection_loss‚0.9609081299806183²ÏÓøï|ü‚(:ñ ©áçÔÂæÞx disk.dm-0.in‚10067.27734375 disk.drbd1260.out‚ 0.52734375$ proc.memory.availableMB‚7.046875% disk./.usageGB‚1156.3607330322266" disk.nvme0n1.in‚10067.27734375 disk.sda.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.95698928833008* proc.memory.percent‚20.516014099121094 cpu‚24.891687380839848( disk.nvme0c0n1.out‚84982.04345703125# disk.dm-0.out‚84982.04345703125 disk.dm-1.in‚0 disk.sda.out‚ 115.5546875 disk.sda3.out‚ 115.5546875 network.recv‚ 1.235039e+06& disk.nvme0n1.out‚84982.04345703125 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 115.5546875 disk.sda3.in‚0 proc.memory.rssMB‚ 3361.34375 proc.cpu.threads‚17 network.sent‚ 5.060908e+06) disk./.usagePercent‚77.60433904507306 disk.sda2.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚10067.27734375èÆÔ¯Žý‚(:ƒ ©áçÔÀð¼y gpu.0.gpu‚4.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“úH$’þ‚(:‡ ¸áçÔØ°ýw gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚116.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üO}ÿ‚(:ò ¸áçÔÄœˆy disk.sda.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 8.76953125* proc.memory.percent‚20.516014099121094 cpu‚24.893151205485808 proc.cpu.threads‚17) disk./.usagePercent‚77.60514162869289$ disk.nvme0c0n1.in‚10067.27734375( disk.nvme0c0n1.out‚85289.46728515625# disk.dm-0.out‚85289.46728515625 disk.sda.out‚ 115.89453125" disk.nvme0n1.in‚10067.27734375& disk.nvme0n1.out‚85289.46728515625 disk.sda2.in‚0 network.sent‚ 5.168313e+06% disk./.usageGB‚1156.3726921081543 disk.dm-0.in‚10067.27734375 disk.dm-1.out‚ 115.89453125 disk.sda3.in‚0 disk.sda3.out‚ 115.89453125 disk.drbd1260.in‚0 disk.drbd1260.out‚0.53125 network.recv‚ 1.240479e+06 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# memory_percent‚99.9464750289917 proc.memory.rssMB‚ 3361.34375#îm€‚(:â ÇáçÔÿàóx$ memory_percent‚99.94652271270752* proc.memory.percent‚20.516014099121094 proc.cpu.threads‚17 network.recv‚ 1.245371e+06 disk.sda3.in‚0 disk.nvme0n1.in‚ 10238.5625& disk.nvme0n1.out‚85783.23681640625 disk.sda2.out‚0& proc.memory.availableMB‚ 8.76171875 cpu‚24.89480515314396 disk.sda.in‚0 disk.sda.out‚ 116.3671875( disk.nvme0c0n1.out‚85783.23681640625 disk.dm-1.in‚0 proc.memory.rssMB‚ 3361.34375$ disk./.usageGB‚1156.372470855713 disk.drbd1260.in‚0 disk.drbd1260.out‚0.53125 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 10238.5625 disk.dm-0.in‚ 10238.5625# disk.dm-0.out‚85783.23681640625 disk.dm-1.out‚ 116.3671875 network.sent‚ 5.174705e+06) disk./.usagePercent‚77.60512678025592 disk.sda3.out‚ 116.3671875 disk.sda2.in‚0 disk.sda1.out‚0ì€dÆœ‚(:‘ ÇáçÔ‰ó“y gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚272.504' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.500800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–y™Vc‚‚(:Ø ÖáçÔÛïþx* proc.memory.percent‚20.516014099121094 cpu‚24.89637816778431 proc.cpu.threads‚17% disk./.usageGB‚1156.3724975585938 disk.nvme0c0n1.in‚ 10238.5625% disk.nvme0c0n1.out‚86129.22265625 disk.dm-0.in‚ 10238.5625$ memory_percent‚99.94649887084961) disk./.usagePercent‚77.60512857230866 disk.nvme0n1.in‚ 10238.5625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.53515625 network.sent‚ 5.179585e+06 network.recv‚ 1.248649e+06# disk.nvme0n1.out‚86129.22265625 disk.sda2.in‚0 disk.dm-0.out‚86129.22265625 disk.dm-1.out‚ 116.703125 disk.sda.out‚ 116.703125$ proc.memory.availableMB‚8.765625 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 116.703125 proc.memory.rssMB‚ 3361.34375Gz¹ƒ‚(:… ÖáçÔ±¬µy gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Î)aœ„‚(:‘ åáçÔ«ùÚx gpu.0.gpu‚58.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.666' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.93319999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`©Ò‰f…‚(:Û åáçÔ£¾’{ disk.drbd1260.in‚0 disk.sda1.out‚0 disk.dm-0.out‚86331.62890625 disk.sda.in‚0$ memory_percent‚99.94652271270752& proc.memory.availableMB‚ 8.76171875 proc.cpu.threads‚17 network.recv‚ 1.253448e+06) disk./.usagePercent‚77.60298322917171% disk./.usageGB‚1156.3405303955078 disk.drbd1260.out‚ 0.53515625# disk.nvme0n1.out‚86331.62890625 disk.nvme0c0n1.in‚ 10238.5625% disk.nvme0c0n1.out‚86331.62890625 disk.dm-0.in‚ 10238.5625 network.sent‚ 5.185771e+06 disk.nvme0n1.in‚ 10238.5625 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 117.140625 disk.sda.out‚ 117.140625 disk.sda3.in‚0 disk.sda3.out‚ 117.140625 disk.sda2.in‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 3361.34375* proc.memory.percent‚20.516014099121094 cpu‚24.897588993191018Óô,v†‚(:ë ôáçÔ’õâ{ disk.sda1.out‚0* proc.memory.percent‚20.516014099121094% disk./.usageGB‚1155.8695449829102( disk.nvme0c0n1.out‚86622.04345703125 disk.dm-0.in‚10238.56640625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 8.7578125 proc.cpu.threads‚17 network.sent‚ 5.190684e+06 disk.sda.out‚ 117.71875 disk.drbd1260.in‚0& disk.nvme0n1.out‚86622.04345703125 disk.sda1.in‚0$ memory_percent‚99.94654655456543 proc.memory.rssMB‚ 3361.34375 cpu‚24.898232706389475 network.recv‚ 1.256755e+06) disk./.usagePercent‚77.57137500294921" disk.nvme0n1.in‚10238.56640625 disk.sda2.in‚0$ disk.nvme0c0n1.in‚10238.56640625# disk.dm-0.out‚86622.04345703125 disk.dm-1.out‚ 117.71875 disk.sda.in‚0 disk.sda3.out‚ 117.71875 disk.drbd1260.out‚ 0.5390625êýFP’‡‚(:‡ ôáçÔ½²„} gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚280.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.0952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W½UÁ2ˆ- _wandbruntime‚3795  _runtime‚3795mV€ØljÁ & lr-AdamW/pg2‚0.0006372689166654198 trainer/global_step‚2249 % _timestamp‚1.7884735884581437e+09 _runtime‚3795.213641965 _step‚179 & lr-AdamW/pg1‚0.0006372689166654198«‹þ7ÌŠÆ & lr-AdamW/pg1‚0.0006372689166654198 & lr-AdamW/pg2‚0.0006372689166654198 trainer/global_step‚2249 % _timestamp‚1.7884735884581437e+09 _runtime‚3795.213641965 _step‚179³#ÊÌ‹2‹- _wandbruntime‚3795  _runtime‚3795-otÿ'ŒùO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352828289618432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326973093425152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55968110 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102913085 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5460336809472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358288652819456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344591516665872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 % _timestamp‚1.7884735884592206e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46944975 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29590373 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352828289618432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344591542093840 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46944975 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5456726149632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46944981 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49924105 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358285015768064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102939429 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55994448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49923367 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319449856007680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7523362656768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 trainer/global_step‚2249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326973218664448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350038701004353 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46944981 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5447184338481 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125239296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _step‚180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352828316009984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5456726149632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚733 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352828316009984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358288652819456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 _runtime‚ 3795.21493426 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712lNp(þO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 _runtime‚ 3795.21493426 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352828289618432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326973093425152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55968110 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102913085 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5460336809472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358288652819456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344591516665872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 % _timestamp‚1.7884735884592206e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46944975 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29590373 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352828289618432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344591542093840 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46944975 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5456726149632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46944981 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49924105 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358285015768064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102939429 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55994448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49923367 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319449856007680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7523362656768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 trainer/global_step‚2249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326973218664448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350038701004353 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46944981 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5447184338481 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125239296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _step‚180 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352828316009984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5456726149632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚733 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352828316009984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358288652819456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0´ÏOÇb2Ž- _wandbruntime‚3796  _runtime‚3796ß/† ƒN MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46965399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29603178 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49945782 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 trainer/global_step‚2249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚716 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5459136631296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319589146396672 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327115823969280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7526677572608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319589095193600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46965399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358444840873472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102984446 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344741871334976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46965405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5449590641767 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56019041 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352982067177472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5459136631296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358444840873472 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5453199304651 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102984446 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 _runtime‚3796.245208141 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350195070639627 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚350191436548775 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55992677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352982093569024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352982093569024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49945066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327115698743296 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327115823969280 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7526677572608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319589095193600 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46965399 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚358444840873472 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102984446 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344741871334976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46965405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5449590641767 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56019041 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352982067177472 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5459136631296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚358444840873472 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5453199304651 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102984446 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 _runtime‚3796.245208141 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350195070639627 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚350191436548775 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55992677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352982093569024 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352982093569024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49945066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327115698743296 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0µW?õA2‘- _wandbruntime‚3796  _runtime‚3796É\’Ö 1loss/SRqqt4/detection_loss‚2.0528572112321855 /loss/FBt4/detection_loss‚2.5980780239288626 _runtime‚3796.246959121 _step‚182 0loss/FRt1/assignment_loss‚2.8802200717441107 0loss/SRqqt3/detection_loss‚2.328634225033425 .loss/FBt2/detection_loss‚1.081934190695248 /loss/FRt3/detection_loss‚3.7908815826688493 /loss/FRt2/assignment_loss‚2.458038519871862 0loss/FBt1/assignment_loss‚0.5763894291594625 trainer/global_step‚2249 % _timestamp‚1.7884735894915464e+09 /loss/FRt1/detection_loss‚1.6078382694809619 epoch‚1 /loss/FRt3/assignment_loss‚7.341202531542097 1loss/SRqqt3/assignment_loss‚2.179226632255155 2loss/SRqqt4/assignment_loss‚2.0455380238592626 1loss/FBt4/assignment_loss‚0.49195016662659824 /loss/FRt2/detection_loss‚2.3808365170108643 .loss/FBt1/detection_loss‚3.634264016151428 /loss/FBt3/detection_loss‚2.4590344791826992 2loss/SRqqt1/assignment_loss‚1.4255888541872666 1loss/SRqqt2/assignment_loss‚1.959052844842275 1loss/FBt3/assignment_loss‚0.49818822273696045 /loss/FRt4/detection_loss‚2.3512977571511757 1loss/SRqqt1/detection_loss‚1.1130379052544996 1loss/SRqqt2/detection_loss‚3.1158241828282676 0loss/FRt4/assignment_loss‚3.7512612500969245 &loss/total_loss‚54.398670276939974 1loss/FBt2/assignment_loss‚0.27749536939953495¦pía“Û /loss/FRt1/detection_loss‚1.6078382694809619 epoch‚1 /loss/FRt3/assignment_loss‚7.341202531542097 1loss/SRqqt3/assignment_loss‚2.179226632255155 2loss/SRqqt4/assignment_loss‚2.0455380238592626 1loss/FBt4/assignment_loss‚0.49195016662659824 /loss/FRt2/detection_loss‚2.3808365170108643 .loss/FBt1/detection_loss‚3.634264016151428 /loss/FBt3/detection_loss‚2.4590344791826992 2loss/SRqqt1/assignment_loss‚1.4255888541872666 1loss/SRqqt2/assignment_loss‚1.959052844842275 1loss/FBt3/assignment_loss‚0.49818822273696045 /loss/FRt4/detection_loss‚2.3512977571511757 1loss/SRqqt1/detection_loss‚1.1130379052544996 1loss/SRqqt2/detection_loss‚3.1158241828282676 0loss/FRt4/assignment_loss‚3.7512612500969245 &loss/total_loss‚54.398670276939974 1loss/FBt2/assignment_loss‚0.27749536939953495 1loss/SRqqt4/detection_loss‚2.0528572112321855 /loss/FBt4/detection_loss‚2.5980780239288626 _runtime‚3796.246959121 _step‚182 0loss/FRt1/assignment_loss‚2.8802200717441107 0loss/SRqqt3/detection_loss‚2.328634225033425 .loss/FBt2/detection_loss‚1.081934190695248 /loss/FRt3/detection_loss‚3.7908815826688493 /loss/FRt2/assignment_loss‚2.458038519871862 0loss/FBt1/assignment_loss‚0.5763894291594625 trainer/global_step‚2249 % _timestamp‚1.7884735894915464e+09¶ÇH/’”‚(:‡ ƒâçÔŽðÁw gpu.0.gpu‚73.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚260.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PUÖh•‚(:Ý ƒâçÔÐ×y disk.nvme0n1.in‚10400.25 disk.sda2.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚10400.25( disk.nvme0c0n1.out‚86910.72314453125 disk.dm-1.out‚ 118.28125$ memory_percent‚99.94809627532959 network.sent‚ 5.210151e+06) disk./.usagePercent‚77.57976129775585 disk.sda.in‚0 disk.sda.out‚ 118.28125 disk.drbd1260.out‚ 0.5390625 disk.dm-0.in‚10400.25& proc.memory.availableMB‚ 8.50390625# proc.memory.rssMB‚ 3361.34765625 network.recv‚ 1.261601e+06% disk./.usageGB‚1155.9945068359375 disk.sda3.in‚0 disk.sda3.out‚ 118.28125 disk.sda1.in‚0# disk.dm-0.out‚86910.72314453125* proc.memory.percent‚20.516037940979004 proc.cpu.threads‚17& disk.nvme0n1.out‚86910.72314453125 disk.sda2.out‚0 disk.dm-1.in‚0 cpu‚24.898978255387135 disk.drbd1260.in‚0mWw*k–‚(:à ’âçÔœ£Õx disk.dm-0.in‚10400.25 disk.sda.out‚ 118.984375& disk.nvme0n1.out‚87342.95166015625& proc.memory.availableMB‚ 8.50390625# proc.memory.rssMB‚ 3361.34765625 cpu‚24.89975332207002( disk.nvme0c0n1.out‚87342.95166015625 disk.dm-1.out‚ 118.984375 disk.drbd1260.in‚0 disk.nvme0n1.in‚10400.25 proc.cpu.threads‚17 disk.sda3.in‚0 network.recv‚ 1.267225e+06) disk./.usagePercent‚77.59503189116575% disk./.usageGB‚1156.2220497131348 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.54296875 disk.sda1.in‚0$ memory_percent‚99.94809627532959# disk.dm-0.out‚87342.95166015625 disk.sda.in‚0 disk.sda3.out‚ 118.984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0* proc.memory.percent‚20.516037940979004 network.sent‚ 5.304619e+06 disk.nvme0c0n1.in‚10400.25V½’—‚(:‡ ’âçÔǯ€y gpu.0.gpu‚61.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚287.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãÒ_Rr˜‚(:ç ¡âçÔ›¤ßx disk.sda2.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.272045e+06# disk.dm-0.out‚87691.63134765625 disk.dm-1.in‚0 disk.nvme0n1.in‚10400.25$ memory_percent‚99.94819164276123# proc.memory.rssMB‚ 3361.34765625 network.sent‚ 5.310788e+06% disk./.usageGB‚1156.2444648742676 disk.dm-0.in‚10400.25 disk.sda.out‚ 119.31640625 disk.sda3.in‚0 disk.sda3.out‚ 119.31640625 disk.sda2.in‚0 disk.nvme0c0n1.in‚10400.25 disk.dm-1.out‚ 119.31640625 disk.drbd1260.out‚ 0.54296875& disk.nvme0n1.out‚87691.63134765625 disk.sda1.out‚0( disk.nvme0c0n1.out‚87691.63134765625& proc.memory.availableMB‚ 8.48828125* proc.memory.percent‚20.516037940979004 cpu‚24.900896575858194) disk./.usagePercent‚77.59653619143695 disk.sda.in‚0 disk.drbd1260.in‚0J€8×™‚(:’ ¡âçÔ“¬y gpu.0.gpu‚56.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚209.128' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.825599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=‹3¡lš‚(:á °âçÔðßx disk.sda.in‚0 disk.drbd1260.out‚0.546875% disk.nvme0n1.out‚87801.4384765625' disk.nvme0c0n1.out‚87801.4384765625 disk.dm-0.in‚10400.25 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.94790554046631& proc.memory.availableMB‚ 8.53515625) disk./.usagePercent‚77.59654566371572 disk.nvme0c0n1.in‚10400.25 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 3361.34765625* proc.memory.percent‚20.516037940979004 proc.cpu.threads‚17" disk.dm-0.out‚87801.4384765625 disk.sda.out‚ 119.69140625 disk.nvme0n1.in‚10400.25 cpu‚24.90260153088028 disk.sda3.out‚ 119.69140625 network.sent‚ 5.315995e+06 network.recv‚ 1.275825e+06% disk./.usageGB‚1156.2446060180664 disk.dm-1.out‚ 119.69140625LEcŠœ›‚(:‘ °âçÔš¥‘y gpu.0.gpu‚45.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.751400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nzÍÊœ‚(:… ¿âçÔ¿°w gpu.0.gpu‚59.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.89' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýv¡|‚(:ñ ¿âçÔñ˹y) disk./.usagePercent‚77.58988690775013$ disk.nvme0c0n1.in‚10504.06640625' disk.nvme0c0n1.out‚88187.2197265625 disk.dm-1.out‚ 120.29296875 disk.drbd1260.out‚0.546875% disk.nvme0n1.out‚88187.2197265625 disk.sda2.in‚0$ memory_percent‚99.94792938232422% disk./.usageGB‚1156.1453857421875 disk.dm-1.in‚0 disk.sda.out‚ 120.29296875 disk.sda3.out‚ 120.29296875 proc.cpu.threads‚17 disk.dm-0.in‚10504.06640625" disk.dm-0.out‚88187.2197265625 disk.sda.in‚0" disk.nvme0n1.in‚10504.06640625 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3361.34765625* proc.memory.percent‚20.516037940979004 network.sent‚ 5.322326e+06 network.recv‚ 1.280743e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0# proc.memory.availableMB‚8.53125 cpu‚24.904095297852514ÁÞÒ2ž- _wandbruntime‚3871  _runtime‚3871üÃê¼ÇŸÁ & lr-AdamW/pg1‚0.0006358448458546981 & lr-AdamW/pg2‚0.0006358448458546981 trainer/global_step‚2299 % _timestamp‚1.7884736650267901e+09 _runtime‚3871.782279149 _step‚183!ªSÌ Æ _runtime‚3871.782279149 _step‚183 & lr-AdamW/pg1‚0.0006358448458546981 & lr-AdamW/pg2‚0.0006358448458546981 trainer/global_step‚2299 % _timestamp‚1.7884736650267901e+09·âîÖ2¡- _wandbruntime‚3871  _runtime‚3871b¤`g¢úO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47966181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚722 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334103276914688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7688901457408 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57226150 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366098069268480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51007773 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 % _timestamp‚1.7884736650277967e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366094432217088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5577264646656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20777155 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334103149578240 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3871.783422754 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚357675491816661 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105165987 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366094432217088 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5571121756665 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352108004150640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360517167570432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStÓ<›!™atsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5577264646656 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326414324254208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57199812 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360517193961984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360517193961984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57226150 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30230623 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20777872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7688825324032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360517167570432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326414375457280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚357679125907305 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ãEäF(£ÿO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326414375457280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚357679125907305 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47966181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚722 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334103276914688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7688901457408 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57226150 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366098069268480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51007773 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 % _timestamp‚1.7884736650277967e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366094432217088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5577264646656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20777155 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334103149578240 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚3871.783422754 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚357675491816661 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105165987 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366094432217088 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5571121756665 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352108004150640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360517167570432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5577264646656 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326414324254208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57199812 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360517193961984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360517193961984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57226150 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30230623 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20777872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7688825324032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360517167570432¸•å?2¤- _wandbruntime‚3872  _runtime‚3872LÛ£Ë '¥„N WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105210978 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360670945129472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 _runtime‚3872.808530044 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47986605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7692125209600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352258307963808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20786771 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5569919397275 trainer/global_step‚2299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105210978 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5579675128320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚357831861451935 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105237348 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105237348 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57224379 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366250620257792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5583285801472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5583285801472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57250743 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57250743 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51030199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334245867175936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360670971521024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30243428 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47986599 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366254257322496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334245739852800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326553614643200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51029480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚352258333391776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 _step‚185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20786057 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57224379 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47986599 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5579675128320 <âl¦‰N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105237348 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57224379 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127323136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366250620257792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5583285801472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5583285801472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57250743 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57250743 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51030199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334245867175936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360670971521024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30243428 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47986599 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚366254257322496 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334245739852800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326553614643200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51029480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚352258333391776 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 _step‚185 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20786057 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57224379 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47986599 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5579675128320 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 LDDeviceStatsMoniîfc‘ tor.on_train_batch_end/inactive_split.large_pool.peak‚877 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105210978 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360670945129472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 _runtime‚3872.808530044 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47986605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7692125209600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚352258307963808 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20786771 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5569919397275 trainer/global_step‚2299 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105210978 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5579675128320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚357831861451935 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105237348 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365312282628969 _runtime‚3948.223129245 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107445233 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚368206071913984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21220742 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 _step‚188 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368206045522432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58431514 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368206071913984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚737 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5691450512173 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 trainer/global_step‚2349 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5701413803520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52091610 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341233496665600 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48987375 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58457852 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7854601758720 % _timestamp‚1.7884737414673564e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341233369329152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚373903848666112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48987381 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107418889 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48987381 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚373907485717504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333378843703808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365315916719613 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚373903848666112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21221474 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719ÐÔ=Ø(¹ÿO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30870868 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48987375 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚373907485717504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5701413803520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58457852 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26338 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7854525625344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚368206045522432 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365312282628969 _runtime‚3948.223129245 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107445233 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚368206071913984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21220742 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 _step‚188 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368206045522432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58431514 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368206071913984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚737 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5691450512173 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 trainer/global_step‚2349 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127336448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5701413803520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52091610 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341233496665600 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48987375 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58457852 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610659840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7854601758720 % _timestamp‚1.7884737414673564e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341233369329152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚373903848666112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48987381 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107418889 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48987381 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚373907485717504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333378843703808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365315916719613 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚373903848666112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76133376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21221474 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719¼ÈEÖ º- _‰¥‰I(wandbruntime‚3949  _runtime‚3949>²Â'»‚N $ _timestamp‚1.788473742505073e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚368359849473024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359774795448576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766896 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚368359823081472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52113322 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333518134092800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374060036706816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359774770020608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5693856815667 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5700213625344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341376103502848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52114033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30883673 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58482445 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341375978276864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58456081 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21229649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107490250 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49007799 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107490250 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5700213625344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374063673771520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49007805 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 _step‚189 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2349 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49007799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58456081 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52114033 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30883673 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58482445 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341375978276864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58456081 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21229649 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107490250 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49007799 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107490250 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5700213625344 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374063673771520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49007805 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 _step‚189 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2349 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49007799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26364 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58456081 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52113322 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333518134092800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374060036706816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359774770020608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5693856815667 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5700213625344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341376103502848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096½Y–œj2½- _wandbruntime‚3949  _runtime‚3949e˨V¾Ð /loss/FRt3/assignment_loss‚7.144968581199646 2loss/SRqqt4/assignment_loss‚2.0607287302132575 0loss/FBt3/assignment_loss‚0.5156797002176366 .loss/FRt4/detection_loss‚2.746235845459474 -loss/FBt1/detection_loss‚4.47794105609258 trainer/global_step‚2349 /loss/FRt4/assignment_loss‚4.080029065544541 1loss/SRqqt1/assignment_loss‚1.377762718512336 1loss/SRqqt2/assignment_loss‚2.207305678299495 2loss/FBt1/assignment_loss‚0.060944684160252414 %loss/total_loss‚55.59664404468097 0loss/FRt1/assignment_loss‚2.5558317031179154 /loss/FRt2/detection_loss‚2.1803912997797683 1loss/SRqqt2/detection_loss‚2.4406806134751866 2loss/SRqqt3/assignment_loss‚1.5142273095746834 0loss/SRqqt4/detection_loss‚2.488604268720073 /loss/FBt2/detection_loss‚0.8505080317272179 _runtime‚3949.263139501 .loss/FRt3/detection_loss‚3.598095786571503 % _timestamp‚1.7884737425075605e+09 1loss/FBt2/assignment_loss‚0.30782359447636404 1loss/SRqqt1/detection_loss‚1.0895884474966584 0loss/SRqqt3/detection_loss‚2.796011745929718 /loss/FBt3/detection_loss‚3.1494739800691605 _step‚190 /loss/FRt1/detection_loss‚1.7277044978775917 /loss/FRt2/assignment_loss‚3.571319195959303 /loss/FBt4/assignment_loss‚0.439867302348526 /loss/FBt4/detection_loss‚2.2149202078580856 epoch‚1Ý'·Q[¿Õ /loss/FRt1/detection_loss‚1.7277044978775917 /loss/FRt2/assignment_loss‚3.571319195959303 /loss/FBt4/assignment_loss‚0.439867302348526 /loss/FBt4/detection_loss‚2.2149202078580856 epoch‚1 /loss/FRt3/assignment_loss‚7.144968581199646 2loss/SRqqt4/assignment_loss‚2.0607287302132575 0loss/FBt3/assignment_loss‚0.5156797002176366 .loss/FRt4/detection_loss‚2.746235845459474 -loss/FBt1/detection_loss‚4.47794105609258 trainer/global_step‚2349 /loss/FRt4/assignment_loss‚4.080029065544541 1loss/SRqqt1/assignment_loss‚1.377762718512336 1loss/SRqqt2/assignment_loss‚2.207305678299495 2loss/FBt1/assignment_loss‚0.060944684160252414 %loss/total_loss‚55.59664404468097 0loss/FRt1/assignment_loss‚2.5558317031179154 /loss/FRt2/detection_loss‚2.1803912997797683 1loss/SRqqt2/detection_loss‚2.4406806134751866 2loss/SRqqt3/assignment_loss‚1.5142273095746834 0loss/SRqqt4/detection_loss‚2.488604268720073 /loss/FBt2/detection_loss‚0.8505080317272179 _runtime‚3949.263139501 .loss/FRt3/detection_loss‚3.598095786571503 % _timestamp‚1.7884737425075605e+09 1loss/FBt2/assignment_loss‚0.30782359447636404 1loss/SRqqt1/detection_loss‚1.0895884474966584 0loss/SRqqt3/detection_loss‚2.796011745929718 /loss/FBt3/detection_loss‚3.1494739800691605 _step‚190¾Ã|ü˜À‚(:„ ™ãçÔ˜™z gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚93.584' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7168 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å©{Á‚(:ð ™ãçÔÝ„{# proc.memory.rssMB‚ 3361.34765625) disk./.usagePercent‚77.63427554209697 disk.dm-1.in‚0 disk.sda3.out‚ 122.90625 disk.drbd1260.in‚0" disk.nvme0n1.in‚10624.09765625 disk.sda2.out‚0$ memory_percent‚99.94862079620361* proc.memory.percent‚20.516037940979004 disk.sda3.in‚0 disk.dm-0.in‚10624.09765625# disk.dm-0.out‚90085.25732421875 disk.dm-1.out‚ 122.90625 disk.sda.in‚0& proc.memory.availableMB‚ 8.41796875 cpu‚24.910332428902233 proc.cpu.threads‚17 network.recv‚ 1.306255e+06 disk.sda.out‚ 122.90625& disk.nvme0n1.out‚90085.25732421875 disk.sda2.in‚0 network.sent‚ 5.457387e+06% disk./.usageGB‚1156.8068084716797$ disk.nvme0c0n1.in‚10624.09765625( disk.nvme0c0n1.out‚90085.25732421875 disk.drbd1260.out‚ 0.55859375 disk.sda1.in‚0 disk.sda1.out‚0_®˜Â‚(:’ ¨ãçÔïž»z gpu.0.gpu‚58.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚220.157' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.031400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^íÕ©uÂ(:ê ¨ãçÔåü€| network.sent‚ 5.564698e+06 network.recv‚ 1.311682e+06% disk.nvme0c0n1.out‚90336.01953125 disk.sda.out‚ 123.26953125 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚10624.09765625 disk.dm-0.in‚10624.09765625 disk.dm-1.out‚ 123.26953125 disk.sda.in‚0 disk.drbd1260.out‚0.5625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-0.out‚90336.01953125 disk.sda3.in‚0" disk.nvme0n1.in‚10624.09765625# disk.nvme0n1.out‚90336.01953125% proc.memory.availableMB‚ 8.4140625# proc.memory.rssMB‚ 3361.34765625 cpu‚24.911633497436313) disk./.usagePercent‚77.63847508968865$ disk./.usageGB‚1156.869384765625 disk.dm-1.in‚0 disk.sda3.out‚ 123.26953125 disk.drbd1260.in‚0 disk.sda1.in‚0$ memory_percent‚99.94864463806152* proc.memory.percent‚20.516037940979004;é `Ä‚(:Õ ·ãçÔʸíx disk.dm-1.in‚0$ memory_percent‚99.94864463806152% disk./.usageGB‚1156.9086647033691 disk.sda.in‚0 disk.sda.out‚ 123.71875" disk.nvme0n1.in‚10624.09765625 disk.nvme0n1.out‚ 90431.3125 disk.sda1.in‚0 disk.dm-1.out‚ 123.71875# proc.memory.rssMB‚ 3361.34765625 network.sent‚ 5.571334e+06 disk.sda3.out‚ 123.71875 disk.sda2.in‚0! disk.nvme0c0n1.out‚ 90431.3125 disk.dm-0.in‚10624.09765625% proc.memory.availableMB‚ 8.4140625 cpu‚24.913099010878785 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚0.5625 disk.sda1.out‚0 disk.dm-0.out‚ 90431.3125* proc.memory.percent‚20.516037940979004 network.recv‚ 1.31643e+06) disk./.usagePercent‚77.64111119926874 disk.sda3.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚10624.09765625Ï°0ÀÅ‚(:… ·ãçÔÑãy gpu.0.gpu‚56.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚201.39' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.278 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M¾¼+Æ‚(:ö ÆãçÔÅÈy disk.drbd1260.out‚ 0.56640625& disk.nvme0n1.out‚90923.33447265625& proc.memory.availableMB‚ 8.41015625# proc.memory.rssMB‚ 3361.34765625 cpu‚24.914383266363796 network.recv‚ 1.319877e+06( disk.nvme0c0n1.out‚90923.33447265625# disk.dm-0.out‚90923.33447265625 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚20.516037940979004 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.out‚ 124.1328125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.94866847991943% disk./.usageGB‚1157.0573806762695$ disk.nvme0c0n1.in‚10758.66015625 disk.dm-0.in‚10758.66015625 disk.dm-1.out‚ 124.1328125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 124.1328125" disk.nvme0n1.in‚10758.66015625 network.sent‚ 5.576209e+06) disk./.usagePercent‚77.65109165299108€ˆéš’Ç‚(:‡ ÆãçÔÞ܇z gpu.0.gpu‚86.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚281.051' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.2102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯åPœÈ‚(:‘ ÕãçÔ×”Œx gpu.0.gpu‚58.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.24' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.248000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™çËFÉ‚(:ô ÕãçÔâîýy network.recv‚ 1.32434e+06 disk.sda1.in‚0$ disk.nvme0c0n1.in‚10758.66015625 disk.dm-1.in‚0 disk.sda3.out‚ 124.53515625 disk.drbd1260.out‚ 0.56640625$ memory_percent‚99.94869232177734 network.sent‚ 5.582304e+06) disk./.usagePercent‚77.64948264563839$ disk./.usageGB‚1157.033405303955" disk.nvme0n1.in‚10758.66015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚10758.66015625# disk.dm-0.out‚91271.20751953125& disk.nvme0n1.out‚91271.20751953125 disk.sda1.out‚0( disk.nvme0c0n1.out‚91271.20751953125 disk.dm-1.out‚ 124.53515625 disk.sda3.in‚0# proc.memory.availableMB‚8.40625# proc.memory.rssMB‚ 3361.34765625* proc.memory.percent‚20.516037940979004 disk.sda.in‚0 disk.sda.out‚ 124.53515625 disk.drbd1260.in‚0 cpu‚24.915783187379922 proc.cpu.threads‚17µg¼‘Ê‚(:† äãçÔ¨¦z gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JêµË‚(:ô äãçÔÞìçz disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.5703125$ memory_percent‚99.94869232177734* proc.memory.percent‚20.516037940979004 cpu‚24.917168139817008& disk.nvme0n1.out‚91617.64404296875 disk.sda1.in‚0( disk.nvme0c0n1.out‚91617.64404296875# disk.dm-0.out‚91617.64404296875 disk.dm-1.in‚0 disk.sda.out‚ 125.08984375 disk.sda3.out‚ 125.08984375# proc.memory.availableMB‚8.40625 network.recv‚ 1.327459e+06% disk./.usageGB‚1157.0567932128906 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3361.34765625( disk./.usagePercent‚77.6510522278308 disk.sda2.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚10758.66015625 disk.dm-0.in‚10758.66015625 disk.dm-1.out‚ 125.08984375 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 5.587296e+06" disk.nvme0n1.in‚10758.66015625š¨d{Ì‚(:ð óãçÔòÙ| disk.sda.in‚0 disk.drbd1260.out‚ 0.5703125& disk.nvme0n1.out‚92060.46044921875* proc.memory.percent‚20.516037940979004) disk./.usagePercent‚77.66783121763245$ disk.nvme0c0n1.in‚10758.66015625 disk.dm-0.in‚10758.66015625 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚99.94866847991943& proc.memory.availableMB‚ 8.41015625 network.recv‚ 1.332297e+06 disk.sda.out‚ 125.484375 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3361.34765625 cpu‚24.91840932943057 proc.cpu.threads‚17 network.sent‚ 5.595446e+06$ disk./.usageGB‚1157.306812286377# disk.dm-0.out‚92060.46044921875 disk.dm-1.in‚0 disk.dm-1.out‚ 125.484375 disk.sda3.out‚ 125.484375 disk.drbd1260.in‚0" disk.nvme0n1.in‚10758.66015625( disk.nvme0c0n1.out‚92060.46044921875ÍãFUÍ‚(:… óãçÔŘä| gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.831' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Æ­2Î- _wandbruntime‚4050  _runtime‚40502JÏÇÏÁ & lr-AdamW/pg1‚0.0006328032546293887 & lr-AdamW/pg2‚0.0006328032546293887 trainer/global_step‚2399 % _timestamp‚1.7884738434003003e+09 _runtime‚4050.157163058 _step‚191¥W¥iÌÐÆ & lr-AdamW/pg2‚0.0006328032546293887 trainer/global_step‚2399 % _timestamp‚1.7884738434003003e+09 _runtime‚4050.157163058 _step‚191 & lr-AdamW/pg1‚0.0006328032546293887¿“¶Ÿ32Ñ- _wandbruntime‚4050  _runtime‚4050%TW©åÒúO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340343414356480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.larg.kp'e_pool.freed‚375894923474432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375894949865984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109698135 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50008575 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340343363153408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚381713265115136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5821952300544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚372952707531921 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59663216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 % _timestamp‚1.7884738434046752e+09 _runtime‚4050.161501957 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21665063 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8020165040128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74036224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21665772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372949073441277 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8020091003904 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50008581 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53176895 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚381716902166528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5811779267681 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚381716902166528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31511118 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348363454157312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50008575 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2399 _step‚192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348363579396608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 % _timestamp‚1.7884738434046752e+09 _runtime‚4050.161501957 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21665063 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8020165040128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74036224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21665772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚372949073441277 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662676 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8020091003904 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090644 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50008581 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53176895 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚381716902166528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5811779267681 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚381716902166528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31511118 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348363454157312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766896 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50008575 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26338 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2399 _step‚192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348363579396608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚381713265115136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5821952300544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610659840À¯8ôÈ2Ô- _wandbruntime‚4052  _runtime‚40524µXÄ 'Õ„N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21673910 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340482704745472 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5810576908291 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109743152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21674599 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109716782 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5820752122368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340482653542400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373101808985699 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376048701033472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50028999 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367291257505376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348506081833984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 _step‚193 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59687783 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53197833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348506207059968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5814185571175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59714147 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚381869453155840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 _runtime‚4052.465970482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50028999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 trainer/global_step‚2399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50029005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚381873090220544 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884738457095456e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376048727425024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376048701033472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5824362795520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5824362795520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53198527 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367291232077408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5820752122368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109743152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364:ý Ö‰N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59714147 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚381869453155840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 _runtime‚4052.465970482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50028999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 trainer/global_step‚2399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50029005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚381873090220544 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884738457095456e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376048727425024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637064704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090852 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637064704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376048701033472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5824362795520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5824362795520 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53198527 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367291232077408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5820752122368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109743152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26364 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21673910 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340482704745472 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662884 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5810576908291 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109743152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21674599 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109716782 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5820752122368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340482653542400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373101808985699 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376048701033472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50028999 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367291257505376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348506081833984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 _step‚193 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59687783 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53197833 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348506207059968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5814185571175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448ÁÛOm‚2×- _wandbruntime‚4052  _runtime‚4052”€HXØÒ /loss/FRt1/detection_loss‚1.7765026483023523 .loss/FRt3/detection_loss‚3.641931692759196 1loss/FBt4/assignment_loss‚0.07493341490836448 % _timestamp‚1.7884738457114706e+09 0loss/FRt4/assignment_loss‚4.6318062464396155 1loss/FBt2/assignment_loss‚0.15897003732427933 0loss/SRqqt4/detection_loss‚2.342904110212584 epoch‚1 _runtime‚4052.466995123 _step‚194 /loss/FRt2/detection_loss‚2.2501378679942143 /loss/FBt1/detection_loss‚3.8047494490941367 /loss/FBt2/detection_loss‚0.8468393957826184 .loss/FBt3/detection_loss‚2.631623851401465 /loss/FRt2/assignment_loss‚2.414038476191069 /loss/FRt3/assignment_loss‚7.190021249983046 0loss/SRqqt2/detection_loss‚2.902776312260401 2loss/SRqqt1/assignment_loss‚1.5485108530035976 1loss/SRqqt4/assignment_loss‚2.148860239499324 1loss/FBt1/assignment_loss‚0.09271400421857834 2loss/SRqqt2/assignment_loss‚2.3100532236553373 .loss/FRt4/detection_loss‚2.539559488164054 1loss/SRqqt3/detection_loss‚2.9780437350273132 %loss/total_loss‚54.89035743705958 1loss/FBt3/assignment_loss‚0.42798344700297875 1loss/SRqqt1/detection_loss‚0.9571409836185701 ,loss/FBt4/detection_loss‚2.9069904088974 trainer/global_step‚2399 0loss/FRt1/assignment_loss‚2.2114914261242924 2loss/SRqqt3/assignment_loss‚2.1017748751947956 y]Ù× % _timestamp‚1.7884738457114706e+09 0loss/FRt4/assignment_loss‚4.6318062464396155 1loss/FBt2/assignment_loss‚0.15897003732427933 0loss/SRqqt4/detection_loss‚2.342904110212584 epoch‚1 _runtime‚4052.466995123 _step‚194 /loss/FRt2/detection_loss‚2.2501378679942143 /loss/FBt1/detection_loss‚3.8047494490941367 /loss/FBt2/detection_loss‚0.8468393957826184 .loss/FBt3/detection_loss‚2.631623851401465 /loss/FRt2/assignment_loss‚2.414038476191069 /loss/FRt3/assignment_loss‚7.190021249983046 0loss/SRqqt2/detection_loss‚2.902776312260401 2loss/SRqqt1/assignment_loss‚1.5485108530035976 1loss/SRqqt4/assignment_loss‚2.148860239499324 1loss/FBt1/assignment_loss‚0.09271400421857834 2loss/SRqqt2/assignment_loss‚2.3100532236553373 .loss/FRt4/detection_loss‚2.539559488164054 1loss/SRqqt3/detection_loss‚2.9780437350273132 %loss/total_loss‚54.89035743705958 1loss/FBt3/assignment_loss‚0.42798344700297875 1loss/SRqqt1/detection_loss‚0.9571409836185701 ,loss/FBt4/detection_loss‚2.9069904088974 trainer/global_step‚2399 0loss/FRt1/assignment_loss‚2.2114914261242924 2loss/SRqqt3/assignment_loss‚2.1017748751947956 /loss/FRt1/detection_loss‚1.7765026483023523 .loss/FRt3/detection_loss‚3.641931692759196 1loss/FBt4/assignment_loss‚0.07493341490836448¿ß'Ú‚(:ô ‚äçÔÉÂýz disk.dm-1.out‚ 125.92578125 proc.cpu.threads‚17 disk.sda2.in‚0) disk./.usagePercent‚77.66783838584341& disk.nvme0n1.out‚92359.29736328125 disk.sda1.out‚0# disk.dm-0.out‚92359.29736328125$ memory_percent‚99.94912147521973% proc.memory.availableMB‚ 8.3359375# proc.memory.rssMB‚ 3361.34765625 network.sent‚ 5.598476e+06 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 10896.7421875( disk.nvme0c0n1.out‚92359.29736328125* proc.memory.percent‚20.516037940979004 cpu‚24.91962569362005 disk.sda2.out‚0 network.recv‚ 1.335613e+06% disk./.usageGB‚1157.3069190979004 disk.sda.in‚0 disk.sda.out‚ 125.92578125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 10896.7421875 disk.sda3.out‚ 125.92578125 disk.drbd1260.out‚ 0.57421875! disk.nvme0n1.in‚ 10896.7421875 disk.dm-1.in‚0]ð6^Û‚(:… ‚äçÔ…æ”| gpu.0.gpu‚57.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚180.01' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ŽèO|Ü‚(:ñ ‘äçÔ€¶y network.sent‚ 5.707165e+06( disk./.usagePercent‚77.6678345457304 disk.dm-0.in‚ 10896.7421875# disk.dm-0.out‚92454.60986328125 disk.dm-1.in‚0 disk.dm-1.out‚ 126.2734375 disk.sda.in‚0 disk.drbd1260.out‚ 0.57421875 network.recv‚ 1.342348e+06% disk./.usageGB‚1157.3068618774414 disk.sda3.in‚0! disk.nvme0n1.in‚ 10896.7421875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚92454.60986328125 disk.sda3.out‚ 126.2734375 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 10896.7421875$ memory_percent‚99.94912147521973% proc.memory.availableMB‚ 8.3359375* proc.memory.percent‚20.516037940979004 proc.cpu.threads‚17 disk.sda.out‚ 126.2734375 disk.drbd1260.in‚0& disk.nvme0n1.out‚92454.60986328125# proc.memory.rssMB‚ 3361.34765625 cpu‚24.920895151646334®HÔa›Ý‚(: ‘äçÔ©y gpu.0.gpu‚4.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.674799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sƒ6¾›Þ‚(:  äçÔÕµx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0')ü"wß‚(:ì  äçÔ™çÇx cpu‚24.929238014123722' disk.nvme0c0n1.out‚92709.6201171875" disk.dm-0.out‚92709.6201171875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 126.68359375% disk.nvme0n1.out‚92709.6201171875$ proc.memory.availableMB‚7.578125 proc.cpu.threads‚17 network.recv‚ 1.345492e+06 disk.sda1.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 10897.3671875 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚1157.309658050537 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 10897.3671875 disk.sda.out‚ 126.68359375 disk.sda3.in‚0( proc.memory.percent‚20.5167293548584 network.sent‚ 5.712001e+06) disk./.usagePercent‚77.66802219925299 disk.dm-0.in‚ 10897.3671875 disk.dm-1.out‚ 126.68359375 disk.drbd1260.out‚ 0.58203125# memory_percent‚99.9537467956543" proc.memory.rssMB‚ 3361.4609375ìÐ$à‚(:’ ¯äçÔЪÕ{ gpu.0.gpu‚87.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘MƆyá‚(:î ¯äçÔïâ«} disk.sda3.out‚ 127.09765625 disk.drbd1260.out‚ 0.58203125 disk.dm-1.out‚ 127.09765625 network.sent‚ 5.718436e+06 disk.sda1.out‚0' disk.nvme0c0n1.out‚93248.5224609375 disk.dm-0.in‚ 10897.3671875 disk.dm-1.in‚0 disk.sda.in‚0( proc.memory.percent‚20.5167293548584 disk.sda.out‚ 127.09765625$ memory_percent‚99.95298385620117 cpu‚24.941853902357042 proc.cpu.threads‚17 network.recv‚ 1.350166e+06% disk./.usageGB‚1157.3096389770508# disk.nvme0c0n1.in‚ 10897.3671875 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 10897.3671875% disk.nvme0n1.out‚93248.5224609375$ proc.memory.availableMB‚7.703125" proc.memory.rssMB‚ 3361.4609375) disk./.usagePercent‚77.66802091921534 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0" disk.dm-0.out‚93248.5224609375 disk.sda3.in‚0È ¬‘â‚(:† ¾äçÔÅøŽy gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚207.786' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O±Spã‚(:å ¾äçԥʰz cpu‚24.95435693073613( disk.nvme0c0n1.out‚93594.40283203125# disk.dm-0.out‚93594.40283203125 disk.dm-1.in‚0& disk.nvme0n1.out‚93594.40283203125 disk.sda1.out‚0" proc.memory.rssMB‚ 3361.4609375) disk./.usagePercent‚77.70157198661519% disk./.usageGB‚1157.8095741271973# disk.nvme0c0n1.in‚ 11053.2578125 disk.sda.out‚127.6875 disk.sda3.out‚127.6875 disk.drbd1260.out‚ 0.5859375 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 5.723384e+06 disk.dm-0.in‚ 11053.2578125 disk.dm-1.out‚127.6875 disk.sda3.in‚0$ memory_percent‚99.95396137237549& proc.memory.availableMB‚ 7.54296875( proc.memory.percent‚20.5167293548584 proc.cpu.threads‚17 network.recv‚ 1.353739e+06 disk.sda.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 11053.2578125 disk.sda2.in‚0c ±\ä‚(:… ÍäçÔÒò–x gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚216.908' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.3816 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*—†¦|å‚(:ñ ÍäçÔ‡¸ãy network.recv‚ 1.358321e+06! disk.nvme0n1.in‚ 11053.2578125# disk.nvme0c0n1.in‚ 11053.2578125 disk.dm-1.in‚0 disk.dm-1.out‚ 128.01953125% disk./.usageGB‚1157.8095741271973 disk.sda2.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚93892.86376953125 disk.dm-0.in‚ 11053.2578125 disk.drbd1260.in‚0& proc.memory.availableMB‚ 7.54296875" proc.memory.rssMB‚ 3361.4609375) disk./.usagePercent‚77.70157198661519 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.out‚ 128.01953125( proc.memory.percent‚20.5167293548584 proc.cpu.threads‚17& disk.nvme0n1.out‚93892.86376953125# disk.dm-0.out‚93892.86376953125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 128.01953125 disk.drbd1260.out‚ 0.5859375$ memory_percent‚99.95396137237549 cpu‚24.96739130375588 network.sent‚ 5.731418e+06ömB¼›æ‚(: ÜäçÔýÉÝx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.323' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.464599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L üúzç‚(:ï ÜäçÔ–Ž¯} disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 11053.2578125( disk.nvme0c0n1.out‚94359.84326171875 disk.sda.in‚0 disk.sda.out‚ 128.3828125 network.recv‚ 1.361547e+06 disk.dm-0.in‚ 11053.2578125# disk.dm-0.out‚94359.84326171875 disk.sda3.out‚ 128.3828125 disk.drbd1260.out‚ 0.58984375$ memory_percent‚99.95248317718506& proc.memory.availableMB‚ 7.78515625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚94359.84326171875" proc.memory.rssMB‚ 3361.4609375( proc.memory.percent‚20.5167293548584 cpu‚24.97828396294284) disk./.usagePercent‚77.70157326665286 disk.sda2.out‚0 disk.dm-1.out‚ 128.3828125 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 11053.2578125 proc.cpu.threads‚17 network.sent‚ 5.736353e+06% disk./.usageGB‚1157.8095932006836¸Þ¶wè‚(:ì ëäçÔÊ©®x disk.sda.out‚ 128.8515625 disk.sda3.in‚0 disk.sda3.out‚ 128.8515625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 11053.2578125 disk.sda2.out‚0& disk.nvme0n1.out‚94455.12060546875( disk.nvme0c0n1.out‚94455.12060546875# disk.dm-0.out‚94455.12060546875$ memory_percent‚99.95245933532715% proc.memory.availableMB‚ 7.7890625" proc.memory.rssMB‚ 3361.4609375( proc.memory.percent‚20.5167293548584 proc.cpu.threads‚17( disk./.usagePercent‚77.7015735226604 disk.dm-1.out‚ 128.8515625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.36655e+06% disk./.usageGB‚1157.8095970153809 disk.sda.in‚0 disk.drbd1260.out‚ 0.58984375# disk.nvme0c0n1.in‚ 11053.2578125 disk.dm-0.in‚ 11053.2578125 cpu‚24.98680099456064 network.sent‚ 5.740873e+06 disk.dm-1.in‚0‰< ‘é‚(:† ëäçÔðôÏy gpu.0.gpu‚79.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚222.795' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.559 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÊzÆê‚(:’ úäçÔú£w gpu.0.gpu‚78.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Ðzë‚(:ï úäçÔšÞy( proc.memory.percent‚20.5167293548584 disk.sda.in‚0 disk.sda.out‚ 129.33203125) disk./.usagePercent‚77.70157224262272 disk.sda1.out‚0# disk.dm-0.out‚94682.90869140625 disk.dm-1.out‚ 129.33203125$ memory_percent‚99.95248317718506 cpu‚24.99606482803319 disk.dm-1.in‚0 disk.drbd1260.in‚0& proc.memory.availableMB‚ 7.78515625 proc.cpu.threads‚17 network.sent‚ 5.745716e+06 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 11248.5546875( disk.nvme0c0n1.out‚94682.90869140625 disk.sda3.out‚ 129.33203125! disk.nvme0n1.in‚ 11248.5546875 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.369627e+06% disk./.usageGB‚1157.8095779418945 disk.dm-0.in‚ 11248.5546875 disk.sda3.in‚0 disk.drbd1260.out‚0.59375& disk.nvme0n1.out‚94682.90869140625" proc.memory.rssMB‚ 3361.4609375í×K2ì- _wandbruntime‚4187  _runtime‚4187|mÝÛ")íâT OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚178 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3719 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚389985659447808 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚380779665019443 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880505153 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5892038426624 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 111743044 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111743044 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚389982926595072 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚384090888168448 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2187 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚22081956 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚347654258938880 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8110846130688 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4507766896 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4509786112 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚389985659447808 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚384090912462848 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚347654218221568 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2706586272 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1532 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚60563261 _step‚195 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2732852736 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚384090888168448 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5894746984960 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚60583967 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2090 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111722333 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20706 ICDeviceStatsMonitoà‡ÝTFr.on_validation_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5884415445171 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚51159077 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚54440954·R’m*îçT YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚54441132 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5894746984960 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1532 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚60583967 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32000 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 25168546816 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 33395048448 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚51159077 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚355765240249344 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚374897979491360 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2732852736 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2708558336 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 135179776 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1532 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2187 trainer/global_step‚2417 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚51159072 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708558336 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2729917088 _runtime‚4187.449945268 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚32359176 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚51159072 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3719 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32000 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚355765064352256 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3719 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111722333 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚389982926595072 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚32359173 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚380782394936531 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3719 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚347654218221568 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2706586272 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1532 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚60563261 _step‚195 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2732852736 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚384090888168448 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5894746984960 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚60583967 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2090 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 111722333 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20706 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5884415445171 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚51159077 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚54440954 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚178 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3719 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚389985659447808 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚380779665019443 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880505153 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5892038426624 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 111743044 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 111743044 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚389982926595072 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚384090888168448 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2187 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚22081956 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚347654258938880 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8110846130688 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4507766896 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4509786112 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚389985659447808 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚384090912462848 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3719 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚390091113682944 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5895297563136 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚179 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1221 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚2090 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚877 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚22086964 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚60571991 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2729917088 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3921 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚347749449289728 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 33395048448 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚32371515 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4509786112 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚390093846535680 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚54458303 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2706586272 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚390093846535680 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 29392819712 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚347749408572416 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚380882960112119 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20706 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚54458482 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚2187 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚32371518 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3719 trainer/global_step‚2417 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚384198524678144 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5895297563136 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1532 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 5884964383095 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚4188.705357332 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚355860981178368 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚2187 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 111746623 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚60592697 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚374997972398208 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2708558336 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚22086788 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507766896 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚355860805281280 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8111531888640 _step‚196 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32000 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8111396708864 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚51174637 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 33395048448 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5892589004800 # _timestamp‚1.78847398194918e+09 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 33395048448 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111746623 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚176 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8111531888640 _step‚196 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚24294400 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32000 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8111396708864 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚51174637 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 33395048448 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5892589004800 # _timestamp‚1.78847398194918e+09 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 33395048448 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 111746623 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚176 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3719 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚390091113682944 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5895297563136 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚179 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1221 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚2090 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚877 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚22086964 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 4509786112 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚60571991 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2729917088 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3921 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚347749449289728 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 33395048448 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚32371515 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4509786112 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚390093846535680 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20711 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚54458303 \MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 2706586272 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚390093846535680 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 29392819712 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚347749408572416 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚380882960112119 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20706 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚54458482 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚2187Ä"q/6 ò‚(:ß ‰åçÔã£Éz disk.sda3.out‚129.625& disk.nvme0n1.out‚95028.43603515625 disk.sda1.out‚0( proc.memory.percent‚20.5167293548584# disk.nvme0c0n1.in‚ 11248.5546875( disk.nvme0c0n1.out‚95028.43603515625 disk.dm-1.in‚0 disk.drbd12-ñ€^60.out‚0.59375 disk.sda1.in‚0$ memory_percent‚99.95427131652832 proc.cpu.threads‚17 disk.dm-1.out‚129.625 disk.sda3.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 7.4921875 cpu‚25.007466700753398% disk./.usageGB‚1157.8097038269043# disk.dm-0.out‚95028.43603515625 disk.sda.in‚0 disk.sda.out‚129.625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 11248.5546875 disk.sda2.out‚0" proc.memory.rssMB‚ 3361.4609375 network.sent‚ 5.751877e+06 network.recv‚ 1.37435e+06) disk./.usagePercent‚77.70158069087135 disk.dm-0.in‚ 11248.5546875vSP›ó‚(: ‰åçÔ›ìøz gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.379199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃF_Woô‚(:ä ˜åçÔŒÅx disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3361.4609375( proc.memory.percent‚20.5167293548584 disk.dm-0.in‚ 11248.5546875 disk.dm-1.out‚130.125 disk.sda.out‚130.125 disk.sda3.in‚0 disk.sda3.out‚130.125! disk.nvme0n1.in‚ 11248.5546875& disk.nvme0n1.out‚95447.29638671875& proc.memory.availableMB‚ 7.71484375 network.recv‚ 1.379747e+06) disk./.usagePercent‚77.70158197090903% disk./.usageGB‚1157.8097229003906# disk.dm-0.out‚95447.29638671875 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd1260.out‚ 0.59765625 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 11248.5546875 disk.sda1.out‚0$ memory_percent‚99.95291233062744 cpu‚25.019127602980678 proc.cpu.threads‚17 network.sent‚ 5.858998e+06( disk.nvme0c0n1.out‚95447.29638671875}i:jõ‚(:„ ˜åçÔÆÔ§y gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚229.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b-úõ2ö- _wandbruntime‚4216  _runtime‚4216™ŸW­v÷¤¥ =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 +particle/sensitivity‚0.2570526897907257 particle/accuracy_7_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3221812844276428 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2151651680469513 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2797481119632721 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1527777761220932 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966413378715515 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.28234028816223145 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 _runtime‚4216.450960041 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.1875 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9791666865348816 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.36419758200645447 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 *jet/accuracy_1_of_6‚0.9230769276618958 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08901155740022659 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.20588235557079315 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9959756135940552 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.27920690178871155 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2255394607782364 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.33527132868766785 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3427983522415161 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0011337868636474013 particle/accuracy_2_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 trainer/global_step‚2417 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.27478665113449097 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3373631536960602 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21157103776931763 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24074076116085052 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06611858308315277 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.27810728549957275 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018468128517270088 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.11363637447357178 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.30666667222976685 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9803443551063538 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12654320895671844 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1303524374961853 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876944422721863 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6751242280006409 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19856178760528564 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.37449511885643005 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9955908060073853 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.01569800078868866 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4062499701976776 particle/accuracy_6_of_7‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16033004224300385 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21867816150188446 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3273293972015381 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2999040186405182 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11472295224666595 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 particle/accuracy_3_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3490234911441803 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07799538969993591 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989420473575592 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0173804871737957 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20906710624694824 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900338649749756 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3505747318267822 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1666666716337204 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22316259145736694 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 epoch‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21628986299037933 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2660714089870453 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17330209910869598 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012039336375892162 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8069957494735718 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4139983355998993 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_2‚0.09307702630758286 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5181323885917664 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3740001916885376 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 +particle/specificity‚0.9556214809417725 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.24244746565818787 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20584502816200256 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_1‚0.3850964903831482 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1513216346502304 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9935243129730225 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22505131363868713 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0163690485060215 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.31578946113586426 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_7_of_7‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1812790483236313 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4775456190109253 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_3‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3118274211883545 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.523809552192688 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.6652520298957825 particle/accuracy_4_of_5‚1 _step‚197 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3647819459438324 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07634659856557846 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1478414535522461 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2703126072883606 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919679164886475 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45092591643333435 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.126227468252182 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1533224880695343 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1239459365606308 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23914140462875366 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1711111217737198 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20701421797275543 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.28234028816223145 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.24347592890262604 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.013759681023657322 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4234059751033783 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23397476971149445 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24239976704120636 8"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.212824285030365 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1782931238412857 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.21568627655506134 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.28732073307037354 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3172153830528259 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22567568719387054 particle/accuracy_5_of_5‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18364198505878448 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21328040957450867 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21168853342533112 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16173270344734192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5423281192779541 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3591388463973999 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006778252311050892 +jet/accuracy_4_of_6‚0.03846153989434242 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22452855110168457 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922491908073425 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.506870687007904 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009860167279839516 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.36950743198394775 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00515872985124588 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4305555522441864 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6364447474479675 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0557003989815712 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41033440828323364 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 % _timestamp‚1.7884740096901193e+09 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20743530988693237 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1612069010734558 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_3_of_4‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21931616961956024 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1953980177640915 jet/accuracy_3_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.20097462832927704 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 jet/accuracy_6_of_6‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9766522645950317 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1940041035413742 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913927316665649 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.026348957791924477 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2550240755081177 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9929699301719666 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19773629307746887 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18745337426662445 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894310235977173 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49501052498817444 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15146751701831818 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7905415892601013 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.28125 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3370506465435028 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3733946681022644 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2753472626209259 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17445805668830872 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18981479108333588 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904637336730957 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9796147346496582 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875923991203308 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24291376769542694 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4446091651916504 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9858649373054504 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 5 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21685691177845 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4551282227039337 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9978963136672974 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3778327405452728 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.422092080116272 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.22863981127738953 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2175438553094864 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17742817103862762 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1979326754808426 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5175755620002747 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5544642210006714 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.22489503026008606 7validation_average_jet_accuracy‚0.36682096123695374 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2567712664604187 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.008219954557716846 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.39605513215065 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15229883790016174 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.34567904472351074 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_7‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.14921678602695465 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4524425268173218 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.283642441034317 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2222222238779068 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8465505242347717 jet/accuracy_2_of_7‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.2368421107530594 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.307583749294281 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5548611283302307 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2083333283662796 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.21394334733486176 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16721153259277344 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3011186718940735 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5438644886016846 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01808447390794754 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.31020674109458923 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_2_of_4‚0.3197273015975952 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222â/7¬þ238779068 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3119793236255646 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.04764748737215996 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 particle/accuracy_1_of_4‚1 'particle/f_score‚0.3432914912700653 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006984014064073563 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.22580234706401825 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5217175483703613 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.27886906266212463 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 )loss/val_total_loss‚56.11981998713256 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2526720464229584 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09574468433856964 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3958333134651184 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5355449318885803 particle/accuracy_2_of_5‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4963458180427551 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.31123995780944824 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1478414535522461 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904165267944336 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 particle/accuracy_1_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2692295014858246 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_2_of_5‚0.3601062595844269 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.35433661937713623 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2872559726238251 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.33172062039375305 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3848913908004761 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1941077560186386 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23439447581768036 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07994559407234192 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011246668174862862 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.8878713250160217 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30747702717781067 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03039460815489292 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3958333134651184 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02586207166314125 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014150943607091904 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2742713689804077 *jet/accuracy_1_of_4‚0.8406063318252563 particle/accuracy_2_of_4‚1 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9231677651405334 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.016282642260193825 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ñEôcø©¥ & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9935243129730225 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22505131363868713 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0163690485060215 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.31578946113586426 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_7_of_7‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1812790483236313 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4775456190109253 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_3‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3118274211883545 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.523809552192688 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.6652520298957825 particle/accuracy_4_of_5‚1 _step‚197 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3647819459438324 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07634659856557846 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1478414535522461 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2703126072883606 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919679164886475 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45092591643333435 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.126227468252182 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1533224880695343 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1239459365606308 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23914140462875366 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1711111217737198 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20701421797275543 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.28234028816223145 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.24347592890262604 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.013759681023657322 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4234059751033783 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23397476971149445 particle/accuracy_1_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24239976704120636 8"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.212824285030365 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1782931238412857 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.21568627655506134 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.28732073307037354 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3172153830528259 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22567568719387054 particle/accuracy_5_of_5‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18364198505878448 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21328040957450867 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21168853342533112 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16173270344734192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5423281192779541 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3591388463973999 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006778252311050892 +jet/accuracy_4_of_6‚0.03846153989434242 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22452855110168457 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922491908073425 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.506870687007904 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 ,jet/accuracy_3_of_6‚0.057692307978868484 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009860167279839516 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.36950743198394775 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00515872985124588 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4305555522441864 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2023809552192688 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6364447474479675 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0557003989815712 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41033440828323364 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 % _timestamp‚1.7884740096901193e+09 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20743530988693237 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1612069010734558 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_3_of_4‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.21931616961956024 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1953980177640915 jet/accuracy_3_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.20097462832927704 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 jet/accuracy_6_of_6‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9766522645950317 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1940041035413742 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9913927316665649 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.026348957791924477 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.2550240755081177 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9929699301719666 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19773629307746887 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18745337426662445 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9894310235977173 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49501052498817444 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15146751701831818 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7905415892601013 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 , Purity/3FRt0SRqqt1FBt/FRt_purity‚0.28125 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3370506465435028 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.3733946681022644 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2753472626209259 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17445805668830872 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6969696879386902 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18981479108333588 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6666666865348816 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904637336730957 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9796147346496582 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9875923991203308 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.24291376769542694 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4446091651916504 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.38257578015327454 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9858649373054504 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 5 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21685691177845 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4551282227039337 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9978963136672974 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3778327405452728 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9991645812988281 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.422092080116272 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.22863981127738953 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2175438553094864 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.17742817103862762 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1979326754808426 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5175755620002747 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5544642210006714 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.22489503026008606 7validation_average_jet_accuracy‚0.36682096123695374 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2567712664604187 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.008219954557716846 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.39605513215065 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.15229883790016174 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.34567904472351074 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_7‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.14921678602695465 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4524425268173218 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.283642441034317 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_5_of_5‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2222222238779068 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8465505242347717 jet/accuracy_2_of_7‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.2368421107530594 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.307583749294281 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5548611283302307 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2083333283662796 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.21394334733486176 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16721153259277344 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3011186718940735 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5438644886016846 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01808447390794754 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.31020674109458923 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_2_of_4‚0.3197273015975952 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9035088419914246 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3119793236255646 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.04764748737215996 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 particle/accuracy_1_of_4‚1 'particle/f_score‚0.3432914912700653 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006984014064073563 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.22580234706401825 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5217175483703613 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.27886906266212463 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 )loss/val_total_loss‚56.11981998713256 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2526720464229584 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09574468433856964 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3958333134651184 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5355449318885803 particle/accuracy_2_of_5‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4963458180427551 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.31123995780944824 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1478414535522461 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904165267944336 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 particle/accuracy_1_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2692295014858246 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_2_of_5‚0.3601062595844269 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.35433661937713623 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.2872559726238251 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.33172062039375305 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3848913908004761 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1941077560186386 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23439447581768036 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07994559407234192 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011246668174862862 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 @&Purity/1g/wļ.FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.8878713250160217 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30747702717781067 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.03039460815489292 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3958333134651184 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02586207166314125 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.014150943607091904 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2742713689804077 *jet/accuracy_1_of_4‚0.8406063318252563 particle/accuracy_2_of_4‚1 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9231677651405334 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.016282642260193825 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 +particle/sensitivity‚0.2570526897907257 particle/accuracy_7_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3221812844276428 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2151651680469513 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2797481119632721 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1527777761220932 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966413378715515 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.28234028816223145 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 _runtime‚4216.450960041 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.1875 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9791666865348816 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.36419758200645447 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 *jet/accuracy_1_of_6‚0.9230769276618958 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08901155740022659 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.20588235557079315 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9959756135940552 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.27920690178871155 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2255394607782364 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.33527132868766785 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3427983522415161 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0011337868636474013 particle/accuracy_2_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 trainer/global_step‚2417 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.27478665113449097 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3373631536960602 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21157103776931763 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.24074076116085052 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06611858308315277 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.27810728549957275 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018468128517270088 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.11363637447357178 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.30666667222976685 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9803443551063538 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12654320895671844 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1303524374961853 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876944422721863 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6751242280006409 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19856178760528564 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.37449511885643005 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9955908060073853 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.01569800078868866 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4062499701976776 particle/accuracy_6_of_7‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16033004224300385 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21867816150188446 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3273293972015381 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2999040186405182 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11472295224666595 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9800000190734863 particle/accuracy_3_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6818181872367859 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3490234911441803 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07799538969993591 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989420473575592 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0173804871737957 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20906710624694824 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 6 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.982758641242981 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900338649749756 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.3505747318267822 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1666666716337204 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22316259145736694 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 epoch‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21628986299037933 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2660714089870453 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17330209910869598 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.012039336375892162 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8069957494735718 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4139983355998993 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_2‚0.09307702630758286 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5181323885917664 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.3740001916885376 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 +particle/specificity‚0.9556214809417725 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.24244746565818787 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20584502816200256 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_1‚0.3850964903831482 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1513216346502304 particle/accuracy_2_of_2‚1Åýr]tù‚(:é §åçÔî½úw disk.sda.out‚ 130.453125 disk.sda3.in‚0! disk.nvme0n1.in‚ 11248.5546875 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚3368.75 network.sent‚ 5.865385e+06 network.recv‚ 1.384642e+06# disk.dm-0.out‚99252.36669921875# memory_percent‚68.2722806930542) proc.memory.percent‚20.56121826171875 cpu‚25.0207956379698 proc.cpu.threads‚17% disk./.usageGB‚1160.3449974060059 disk.dm-0.in‚ 11248.5546875 disk.dm-1.in‚0 disk.dm-1.out‚ 130.453125 disk.drbd1260.in‚0( disk.nvme0c0n1.out‚99252.36669921875 disk.sda.in‚0 disk.sda3.out‚ 130.453125 disk.drbd1260.out‚ 0.59765625& disk.nvme0n1.out‚99252.36669921875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 5198.26953125) disk./.usagePercent‚77.87172637021786# disk.nvme0c0n1.in‚ 11248.5546875*géú‚(:„ §åçÔ’Ûˆy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ñ½h‘û‚(:† ¶åçÔ†ˆ‰y gpu.0.gpu‚17.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!÷ûyü‚(:ô ¶åçÔ ÐÒy% disk./.usageGB‚1161.8176040649414 disk.dm-0.in‚11303.55859375$ disk.dm-0.out‚104437.16845703125 disk.sda.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚104437.16845703125$ memory_percent‚68.26882362365723 network.recv‚ 1.389406e+06) disk.nvme0c0n1.out‚104437.16845703125 disk.dm-1.in‚0 disk.sda.out‚ 130.8203125 disk.sda3.out‚ 130.8203125" disk.nvme0n1.in‚11303.55859375 disk.sda2.out‚0 disk.sda1.out‚0) disk./.usagePercent‚77.97055423869831$ disk.nvme0c0n1.in‚11303.55859375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.60546875 disk.sda2.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚3368.75 cpu‚25.020242985616015 disk.dm-1.out‚ 130.8203125( proc.memory.availableMB‚ 5198.8359375) proc.memory.percent‚20.56121826171875 proc.cpu.threads‚17 network.sent‚ 5.987558e+06`—¢ÿ‘ý‚(:† ÅåçÔ°–Ÿ~ gpu.0.gpu‚23.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬^lD‚þ‚(:÷ ÅåçÔ¥úÞ~ network.recv‚ 1.393971e+06" disk.nvme0n1.in‚11388.37890625 disk.sda1.out‚0 disk.dm-0.in‚11388.37890625 disk.sda.in‚0 disk.sda3.out‚ 131.23046875 disk.drbd1260.in‚0' disk.nvme0n1.out‚105073.32470703125 disk.sda2.in‚0) disk.nvme0c0n1.out‚105073.32470703125 disk.drbd1260.out‚ 0.60546875) proc.memory.percent‚20.56121826171875 disk.sda2.out‚0$ disk.dm-0.out‚105073.32470703125$ memory_percent‚68.26882362365723( proc.memory.availableMB‚ 5198.8359375 proc.memory.rssMB‚3368.75 cpu‚25.021145388809586 proc.cpu.threads‚17) disk./.usagePercent‚77.97001636686888% disk./.usageGB‚1161.8095893859863 disk.sda1.in‚0$ disk.nvme0c0n1.in‚11388.37890625 disk.dm-1.in‚0 disk.dm-1.out‚ 131.23046875 disk.sda.out‚ 131.23046875 disk.sda3.in‚0 network.sent‚ 5.993768e+06ÿ)lÍ€ÿ‚(:õ ÔåçÔé½òw' disk.nvme0n1.out‚105348.19384765625 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 1.397228e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.609375 disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 5198.67578125 disk.sda3.out‚ 131.58984375 network.sent‚ 5.998605e+06( disk./.usagePercent‚77.9684426885559$ disk.dm-0.out‚105348.19384765625 disk.dm-1.out‚ 131.58984375 disk.sda.in‚0 disk.sda.out‚ 131.58984375" disk.nvme0n1.in‚11388.37890625% disk./.usageGB‚1161.7861404418945) disk.nvme0c0n1.out‚105348.19384765625 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚68.26980113983154 proc.memory.rssMB‚3368.75) proc.memory.percent‚20.56121826171875 cpu‚25.022126640415877$ disk.nvme0c0n1.in‚11388.37890625 disk.dm-0.in‚11388.37890625ÂèK¢€‚(:… ÔåçÔ—Äx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.006' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6012 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0% €‚(:õ ãåçÔ­¹‰y disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚68.26436519622803! proc.memory.rssMB‚ 3367.765625) disk./.usagePercent‚77.97001636686888 disk.sda2.in‚0) proc.memory.percent‚20.55521011352539 proc.cpu.threads‚17 network.recv‚ 1.402415e+06% disk./.usageGB‚1161.8095893859863) disk.nvme0c0n1.out‚105765.92822265625$ disk.dm-0.out‚105765.92822265625 disk.dm-1.in‚0 disk.dm-1.out‚ 131.8984375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.609375 network.sent‚ 6.005041e+06$ disk.nvme0c0n1.in‚11388.37890625 disk.sda3.out‚ 131.8984375" disk.nvme0n1.in‚11388.37890625' disk.nvme0n1.out‚105765.92822265625 disk.sda1.in‚0) proc.memory.availableMB‚ 5199.56640625 cpu‚25.0226069910764 disk.dm-0.in‚11388.37890625 disk.sda.in‚0 disk.sda.out‚ 131.8984375 disk.sda3.in‚06ß÷Ò‚‚(:… ãåçÔ¶¡Ÿy gpu.0.gpu‚60.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚216.39' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.278 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[êÖÞ2ƒ- _wandbruntime‚4304  _runtime‚4304Òt‡9Æ„À $ _timestamp‚1.788474098028155e+09 _runtime‚4304.785328056 _step‚198 & lr-AdamW/pg1‚0.0006311863680778532 & lr-AdamW/pg2‚0.0006311863680778532 trainer/global_step‚2449Ñ!ÂgË…Å $ _timestamp‚1.788474098028155e+09 _runtime‚4304.785328056 _step‚198 & lr-AdamW/pg1‚0.0006311863680778532 & lr-AdamW/pg2‚0.0006311863680778532 trainer/global_step‚2449ÆÇs}2†- _wandbruntime‚4304  _runtime‚4304˜S¾(‡úO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5977194211328 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 % _timestamp‚1.7884740980314941e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386925121513647 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚390364881309696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390364854918144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52010055 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55353998 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380958413293872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8224194606592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5963099557095 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52010055 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 118947328 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61446840 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52010061 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5977194211328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 _runtime‚4304.788517571 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396342075521024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _step‚199 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32928977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396338438469120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8224262350848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380958387865904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚396342075521024 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52010061 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390364854918144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32928972 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5973583550976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22425026 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386921487422999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55354723 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361529218496512 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5973583550976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22425746 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396338438469120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361529099549184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353304904942592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67744256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024||Œ»’ ˆÿO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚390364881309696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390364854918144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52010055 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55353998 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380958413293872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8224194606592 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5963099557095 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52010055 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 118947328 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61446840 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52010061 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5977194211328 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 _runtime‚4304.788517571 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396342075521024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _step‚199 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32928977 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396338438469120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8224262350848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_byteï,ss.all.current‚ 3634090648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚380958387865904 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚396342075521024 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52010061 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚390364854918144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32928972 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5973583550976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22425026 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚386921487422999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55354723 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361529218496512 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5973583550976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22425746 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396338438469120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361529099549184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353304904942592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67744256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5977194211328 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 % _timestamp‚1.7884740980314941e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚386925121513647ÇrHÛ‰‚(:ö òåçÔÞû«z( disk.nvme0c0n1.out‚106161.7646484375 disk.dm-1.in‚0 disk.sda3.out‚ 132.44140625 disk.drbd1260.out‚ 0.61328125" disk.nvme0n1.in‚11388.37890625' proc.memory.availableMB‚ 5199.296875 cpu‚25.023082907736956 disk.dm-0.in‚11388.37890625 disk.sda.out‚ 132.44140625 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 6.00998e+06$ disk.nvme0c0n1.in‚11388.37890625# disk.dm-0.out‚106161.7646484375 disk.dm-1.out‚ 132.44140625& disk.nvme0n1.out‚106161.7646484375$ memory_percent‚68.26601028442383) disk./.usagePercent‚77.96661530677649% disk./.usageGB‚1161.7589111328125 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 3367.765625) proc.memory.percent‚20.55521011352539 network.recv‚ 1.405826e+06œÏ1\‘Š‚(:† òåçÔ•’ž{ gpu.0.gpu‚12.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.433' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05›‘I2‹- _wandbruntime‚4307  _runtime‚43074Ö¦ƒ 'Œ„N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61497772 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390518658868736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113528257 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113501886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67730944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387077857058277 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32941777 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361671741864448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8227546532864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969114523269 % _timestamp‚1.7884741002897744e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113528257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚396498263575040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22434658 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381108717107040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55376440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387074222967421 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113501886 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 118934016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5965505860381 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52030485 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61497772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32941782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22433983 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8227614263808 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚396498263575040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353444246534656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 trainer/global_step‚2449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52030479 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5979604706304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61471407 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5979604706304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5975994032640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55375760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 _step‚200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚02Û'‰N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚67730944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387077857058277 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32941777 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361671741864448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8227546532864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969114523269 % _timestamp‚1.7884741002897744e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113528257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚396498263575040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22434658 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚381108717107040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55376440 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387074222967421 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113501886 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 118934016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5965505860381 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52030485 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61497772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32941782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22433983 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8227614263808 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚396498263575040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚353444246534656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 trainer/global_step‚2449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52030479 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5979604706304 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61471407 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5979604706304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5975994032640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55375760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 _step‚200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61497772 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390518658868736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113528257 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113501886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675È©:–2Ž- _wandbruntime‚4307  _runtime‚4307»€ð[Õ /loss/FBt2/detection_loss‚1.0209096205759456 &loss/total_loss‚54.995521871635844 2loss/SRqqt4/assignment_loss‚2.0171262201140907 0loss/FBt4/assignment_loss‚0.3605040293800365 /loss/FRt1/detection_loss‚1.6696440598364763 .loss/FRt3/detection_loss‚4.066260238488515 1loss/SRqqt2/detection_loss‚2.6133043503761293 .loss/FBt1/detection_loss‚3.729368656873703 0loss/FRt1/assignment_loss‚2.3067137632105084 0loss/FRt2/assignment_loss‚2.6804743664605275 0loss/FRt4/assignment_loss‚3.8734318324497767 2loss/SRqqt3/assignment_loss‚1.8445896609037211 0loss/FBt1/assignment_loss‚0.7313915397971869 /loss/FRt2/detection_loss‚2.1465868190640496 /loss/FRt4/detection_loss‚2.8075190578188214 /loss/FRt3/assignment_loss‚7.451771537462871 trainer/global_step‚2449 1loss/SRqqt3/detection_loss‚2.5095675858584316 0loss/FBt3/assignment_loss‚0.1586593250659379 .loss/FBt4/detection_loss‚2.526148787566594 _step‚201 2loss/SRqqt1/assignment_loss‚1.3130953993822914 1loss/FBt2/assignment_loss‚0.20060625280419822 1loss/SRqqt1/detection_loss‚0.9913019044906832 0loss/SRqqt4/detection_loss‚2.690605482634376 .loss/FBt3/detection_loss‚3.340600685639815 epoch‚2 % _timestamp‚1.7884741002989378e+09 2loss/SRqqt2/assignment_loss‚1.9453406953811645 _runtime‚4307.055165463+G×s`Ú 1loss/SRqqt2/detection_loss‚2.6133043503761293 .loss/FBt1/detection_loss‚3.729368656873703 0loss/FRt1/assignment_loss‚2.3067137632105084 0loss/FRt2/assignment_loss‚2.6804743664605275 0loss/FRt4/assignment_loss‚3.8734318324497767 2loss/SRqqt3/assignment_loss‚1.8445896609037211 0loss/FBt1/assignment_loss‚0.7313915397971869 /loss/FRt2/detection_loss‚2.1465868190640496 /loss/FRt4/detection_loss‚2.8075190578188214 /loss/FRt3/assignment_loss‚7.451771537462871 trainer/global_step‚2449 1loss/SRqqt3/detection_loss‚2.5095675858584316 0loss/FBt3/assignment_loss‚0.1586593250659379 .loss/FBt4/detection_loss‚2.526148787566594 _step‚201 2loss/SRqqt1/assignment_loss‚1.3130953993822914 1loss/FBt2/assignment_loss‚0.20060625280419822 1loss/SRqqt1/detection_loss‚0.9913019044906832 0loss/SRqqt4/detection_loss‚2.690605482634376 .loss/FBt3/detection_loss‚3.340600685639815 epoch‚2 % _timestamp‚1.7884741002989378e+09 2loss/SRqqt2/assignment_loss‚1.9453406953811645 _runtime‚4307.055165463 /loss/FBt2/detection_loss‚1.0209096205759456 &loss/total_loss‚54.995521871635844 2loss/SRqqt4/assignment_loss‚2.0171262201140907 0loss/FBt4/assignment_loss‚0.3605040293800365 /loss/FRt1/detection_loss‚1.6696440598364763 .loss/FRt3/detection_loss‚4.066260238488515É^{™Ž‘‚(:ƒ æçÔòùßz gpu.0.gpu‚4.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°¹¬{’‚(:ð æçÔýà{( disk./.usagePercent‚77.9666176108443$ disk./.usageGB‚1161.758945465088) disk.nvme0c0n1.out‚106546.65771484375 disk.sda.out‚ 132.91015625 disk.drbd1260.out‚ 0.61328125' disk.nvme0n1.out‚106546.65771484375! disk.nvme0c0n1.in‚ 11522.59375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 3367.765625 cpu‚25.02389379930267 disk.dm-0.in‚ 11522.59375 disk.dm-1.out‚ 132.91015625 disk.sda3.out‚ 132.91015625 disk.nvme0n1.in‚ 11522.59375) proc.memory.availableMB‚ 5199.30078125 proc.cpu.threads‚17$ disk.dm-0.out‚106546.65771484375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚68.26598644256592) proc.memory.percent‚20.55521011352539 network.sent‚ 6.107311e+06 network.recv‚ 1.412319e+06ý–› ’“‚(:‡ æçÔ‡þÐx gpu.0.gpu‚72.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚255.123' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žê¼ƒM”‚(:ð æçÔÔòØ{ disk.dm-0.in‚ 11522.59375$ disk.dm-0.out‚1068v{Ÿ.774.30322265625 disk.sda2.in‚0' proc.memory.availableMB‚ 5199.296875 network.recv‚ 1.417463e+06% disk./.usageGB‚1161.7589645385742 disk.dm-1.in‚0 disk.sda.out‚ 133.18359375 disk.sda3.out‚ 133.18359375$ memory_percent‚68.26601028442383! proc.memory.rssMB‚ 3367.765625) proc.memory.percent‚20.55521011352539! disk.nvme0c0n1.in‚ 11522.59375) disk.nvme0c0n1.out‚106774.30322265625 disk.sda.in‚0 disk.sda1.out‚0 cpu‚25.024772787838163 proc.cpu.threads‚17 disk.dm-1.out‚ 133.18359375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6171875 disk.nvme0n1.in‚ 11522.59375' disk.nvme0n1.out‚106774.30322265625 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 6.204209e+06) disk./.usagePercent‚77.96661889088196ZÕžm•‚(:’ ŸæçÔÍ®Áy gpu.0.gpu‚80.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.342' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.868399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íwTy–‚(:î ŸæçÔ‚ÿªz cpu‚25.02560451240619 network.recv‚ 1.422321e+06! disk.nvme0c0n1.in‚ 11522.59375 disk.dm-0.in‚ 11522.59375 disk.sda.out‚ 133.6640625 disk.sda3.in‚0 disk.nvme0n1.in‚ 11522.59375 disk.sda2.in‚0) disk./.usagePercent‚77.96661786685183% disk./.usageGB‚1161.7589492797852 disk.sda3.out‚ 133.6640625' disk.nvme0n1.out‚107096.91650390625$ memory_percent‚68.26622486114502) proc.memory.availableMB‚ 5199.26171875! proc.memory.rssMB‚ 3367.765625 network.sent‚ 6.210529e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 133.6640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6171875) proc.memory.percent‚20.55521011352539 proc.cpu.threads‚17) disk.nvme0c0n1.out‚107096.91650390625$ disk.dm-0.out‚107096.91650390625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda2.out‚0Oö"Aw—‚(:ì ®æçÔ™Ðéx! proc.memory.rssMB‚ 3367.765625) proc.memory.percent‚20.55521011352539 disk.drbd1260.in‚0$ memory_percent‚68.26629638671875 cpu‚25.026544892184475 proc.cpu.threads‚17 network.recv‚ 1.425447e+06 disk.dm-0.in‚ 11522.59375 disk.sda.out‚ 134.00390625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.62109375 disk.nvme0n1.in‚ 11522.59375' disk.nvme0n1.out‚107347.59521484375 disk.sda2.out‚0 network.sent‚ 6.21558e+06) disk.nvme0c0n1.out‚107347.59521484375 disk.dm-1.in‚0 disk.dm-1.out‚ 134.00390625 disk.sda3.out‚ 134.00390625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚77.96660557849019% disk./.usageGB‚1161.7587661743164! disk.nvme0c0n1.in‚ 11522.59375$ disk.dm-0.out‚107347.59521484375 disk.sda.in‚0# proc.memory.availableMB‚5199.25wc(œ˜‚(:‘ ®æçÔ‘ø‰z gpu.0.gpu‚36.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚212.878' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.575599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áƒ™‚(:ø ½æçÔûåÄy disk.sda1.out‚0$ disk.nvme0c0n1.in‚11682.39453125 disk.dm-0.in‚11682.39453125 disk.dm-1.in‚0 disk.sda3.in‚0! proc.memory.rssMB‚ 3367.765625 network.sent‚ 6.221707e+06% disk./.usageGB‚1161.7587585449219 disk.sda1.in‚0$ disk.dm-0.out‚107513.95068359375 disk.dm-1.out‚ 134.4296875 disk.sda.out‚ 134.4296875 disk.drbd1260.in‚0 network.recv‚ 1.430097e+06 disk.drbd1260.out‚ 0.62109375" disk.nvme0n1.in‚11682.39453125$ memory_percent‚68.26786994934082) proc.memory.percent‚20.55521011352539 cpu‚25.027508720650584 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚107513.95068359375 disk.sda.in‚0 disk.sda3.out‚ 134.4296875' disk.nvme0n1.out‚107513.95068359375( proc.memory.availableMB‚ 5198.9921875 proc.cpu.threads‚17) disk./.usagePercent‚77.96660506647513Ù?wR‘š‚(:† ½æçÔžÝy gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.567' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä´Í’›‚(:‡ ÌæçÔ³Õàx gpu.0.gpu‚53.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚204.318' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.8636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø& 1}œ‚(:ò ÌæçÔ†Ûíy disk.sda2.in‚0 disk.sda.in‚0" disk.nvme0n1.in‚11682.39453125' disk.nvme0n1.out‚107920.81591796875) disk.nvme0c0n1.out‚107920.81591796875 disk.dm-0.in‚11682.39453125 disk.dm-1.in‚0$ memory_percent‚68.26679706573486) proc.memory.availableMB‚ 5199.16796875" disk./.usageGB‚1161.7587890625 disk.sda2.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚11682.39453125 disk.dm-1.out‚ 134.91015625! proc.memory.rssMB‚ 3367.765625) proc.memory.percent‚20.55521011352539 cpu‚25.02830394928619) disk./.usagePercent‚77.96660711453539 disk.sda1.in‚0$ disk.dm-0.out‚107920.81591796875 proc.cpu.threads‚17 network.sent‚ 6.22836e+06 network.recv‚ 1.433426e+06 disk.sda.out‚ 134.91015625 disk.sda3.in‚0 disk.sda3.out‚ 134.91015625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.625µŠê62- _wandbruntime‚4406  _runtime‚4406kyKÇžÁ _step‚202 & lr-AdamW/pg1‚0.0006295058657334901 & lr-AdamW/pg2‚0.0006295058657334901 trainer/global_step‚2499 % _timestamp‚1.7884742001517177e+09 _runtime‚4406.907759841úËYdÌŸÆ _step‚202 & lr-AdamW/pg1‚0.0006295058657334901 & lr-AdamW/pg2‚0.0006295058657334901 trainer/global_step‚2499 % _timestamp‚1.7884742001517177e+09 _runtime‚4406.907759841Êñ:2 - _wandbruntime‚4406  _runtime‚4406“i7(¡úO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚404151491970048 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚398053732870144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69841408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398053759261696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53031255 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33569227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388474875350672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388474849922704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62704881 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394558278235307 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404147854918144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22875247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8391437578752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6094122048000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398053759261696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394561912325955 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚4406.909003435 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6097732708352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 _step‚203 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚404147854918144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360269424392192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚368660913174016 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62704881 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33569222 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8391367737344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚404151491970048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6097732708352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121044480 % _timestamp‚1.7884742001526203e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53031261 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53031255 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115736142 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚398053759261696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53031255 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33569227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚388474875350672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚388474849922704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62704881 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394558278235307 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404147854918144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22875247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8391437578752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6094122048000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398053759261696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394561912325955 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚4406.909003435 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6097732708352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 _step‚203 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚404147854918144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360269424392192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚368660913174016 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62704881 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33569222 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8391367737344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚404151491970048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6097732708352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121044480 % _timestamp‚1.7884742001526203e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53031261 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53031255 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115736142 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53051685 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6100143203328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56466213 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388625179163840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6096532529664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115781159 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360408714781184 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115781159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368803447725568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394714647870585 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115754788 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398207510429184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884742015730045e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62729474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_endæPhÂ)/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6085834615889 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398207536820736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22883477 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62703109 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6096532529664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33582032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394711013779729 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56465504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69828096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53051679 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6089443278777 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22884181 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚368803568756736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚404307680024064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8394732944384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398207536820736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62729474 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53051685 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6100143203328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56466213 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚388625179163840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6096532529664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115781159 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360408714781184 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115781159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368803447725568 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394714647870585 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115754788 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398207510429184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884742015730045e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62729474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6085834615889 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398207536820736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22883477 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62703109 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6096532529664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33582032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394711013779729 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56465504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69828096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53051679 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6089443278777 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22884181 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚368803568756736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚404307680024064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8394732944384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398207536820736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62729474 P~µ‚(:ó µççÔ¦¶÷w network.sent‚ 6.369085e+06 network.recv‚ 1.464884e+06) disk./.usagePercent‚77.96818412094632% disk./.usageGB‚1161.7822875976562 disk.sda1.in‚0( disk.nvme0c0n1.out‚110181.7158203125 disk.sda.out‚ 138.13671875 disk.sda3.out‚ 138.13671875 disk.drbd1260.in‚0& disk.nvme0n1.out‚110181.7158203125 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 13032.765625 disk.dm-0.in‚ 13032.765625 disk.dm-1.out‚ 138.13671875 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.63671875 disk.nvme0n1.in‚ 13032.765625 disk.dm-1.in‚0) proc.memory.availableMB‚ 1486.75390625! proc.memory.rssMB‚ 3367.765625 disk.sda2.out‚0 disk.sda1.out‚0# disk.dm-0.out‚110181.7158203125$ memory_percent‚90.92557430267334) proc.memory.percent‚20.55521011352539 cpu‚25.033499265680906 proc.cpu.threads‚17îz ›¶‚(: µççÔ£€›x gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚130.922' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.184400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+·‚(:’ ÄççÔÿÉÔx gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.977' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.395399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìoñèu¸‚(:ê ÄççÔß±á{ disk.drbd1260.in‚0 network.recv‚ 1.468407e+06) disk./.usagePercent‚77.96818565699152% disk./.usageGB‚1161.7823104858398 disk.sda2.in‚0 disk.dm-1.out‚ 138.78125 disk.sda.in‚0# memory_percent‚90.9245252609253) proc.memory.percent‚20.55521011352539" disk.nvme0c0n1.in‚ 13032.765625$ disk.dm-0.out‚110456.51220703125 disk.dm-1.in‚0 disk.drbd1260.out‚0.640625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 13032.765625) disk.nvme0c0n1.out‚110456.51220703125 disk.dm-0.in‚ 13032.765625 disk.sda3.out‚ 138.78125) proc.memory.availableMB‚ 1486.92578125! proc.memory.rssMB‚ 3367.765625 cpu‚25.034268816957674 network.sent‚ 6.374072e+06' disk.nvme0n1.out‚110456.51220703125 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 138.78125 disk.sda3.in‚0y5\2¹- _wandbruntime‚4517  _runtime‚4517ÛqšØǺÁ & lr-AdamW/pg1‚0.0006277620978104692 & lr-AdamW/pg2‚0.0006277620978104692 trainer/global_step‚2549 % _timestamp‚1.7884743111842654e+09 _runtime‚4517.940003551 _step‚206¹ó<Ì»Æ & lr-AdamW/pg2‚0.0006277620978104692 trainer/global_step‚2549 % _timestamp‚1.7884743111842654e+09 _runtime‚4517.940003551 _step‚206 & lr-AdamW/pg1‚0.0006277620978104692Îå¬÷Ñ2¼- _wandbruntime‚4517  _runtime‚4517ÜuŽF”½úO _step‚207 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6203757068111 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54052461 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6218271205376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚405742610822144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63910244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54052455 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411957271367168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54052455 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69841408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411960908419072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117962699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6214660545024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6218271205376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚405742637213696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34209477 trainer/global_step‚2549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57530719 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚405742637213696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4517.941522384ŠDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411957271367168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54052455 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69841408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚411960908419072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117962699 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6214660545024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6218271205376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚405742637213696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34209477 trainer/global_step‚2549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57530719 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚405742637213696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4517.941522384 _step‚207 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6203757068111 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54052461 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6218271205376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚405742610822144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63910244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54052455 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 Jîl! 'À„N XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6206163371397 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412117096473088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375935369847808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8562014585856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412117096473088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54072885 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚396141666648608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚396141641220640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4519.105422112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚405896388381184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34222282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6217071026688 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚405896414772736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6217071026688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63934811 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402347804592037 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 _step‚208 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375935248816640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367373234230784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54072879 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6220681700352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57552974 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23330009 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54072885 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57552286 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚405896388381184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121031168 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367373285433856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6220681700352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412113459407872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118034061 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 % _timestamp‚1.7884743123482707e+09Q1»0Á‰N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402347804592037 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 _step‚208 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375935248816640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367373234230784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54072879 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6220681700352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57552974 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23330009 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54072885 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57552286 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚405896388381184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121031168 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367373285433856 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6220681700352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 WHDeviceStatsMonitor.on_train_batch_end/request±7Žß ed_bytes.small_pool.current‚ 3608662888 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412113459407872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118034061 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 % _timestamp‚1.7884743123482707e+09 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6206163371397 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412117096473088 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375935369847808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8562014585856 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412117096473088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54072885 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚396141666648608 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚396141641220640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4519.105422112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚405896388381184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34222282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6217071026688 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚405896414772736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6217071026688 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63934811ЂP¿f2Â- _wandbruntime‚4519  _runtime‚4519Ÿ¥]^WÃÑ /loss/FRt4/assignment_loss‚4.639316244823177 2loss/SRqqt3/assignment_loss‚2.3283715265638687 0loss/FBt3/assignment_loss‚0.2698384848263231 1loss/SRqqt2/detection_loss‚2.6541971219213387 trainer/global_step‚2549 # _timestamp‚1.78847431235035e+09 _runtime‚4519.107303257 1loss/FBt2/assignment_loss‚0.22588958367414205 /loss/FRt2/detection_loss‚2.3123105244178856 1loss/SRqqt1/detection_loss‚0.9174248453516226 /loss/FBt1/detection_loss‚2.7438236847519875 /loss/FRt3/assignment_loss‚6.104186021364653 2loss/SRqqt2/assignment_loss‚2.2540104608786735 2loss/SRqqt4/assignment_loss‚2.4775215858413326 1loss/FBt4/assignment_loss‚0.36077701678389973 1loss/SRqqt3/detection_loss‚2.4351040808593525 /loss/FBt2/detection_loss‚0.8791857281433685 _step‚209 -loss/FRt3/detection_loss‚3.94843813089224 /loss/FBt3/detection_loss‚3.0609240423549307 %loss/total_loss‚54.55905917590751 /loss/FRt1/assignment_loss‚2.825069866861616 2loss/SRqqt1/assignment_loss‚1.4738335005939007 .loss/FRt2/assignment_loss‚3.32616563921883 /loss/FRt1/detection_loss‚1.6579696829698929 /loss/FRt4/detection_loss‚2.6850157394641783 0loss/SRqqt4/detection_loss‚2.484635385774797 .loss/FBt4/detection_loss‚2.430594325065613 epoch‚2 1loss/FBt1/assignment_loss‚0.06445595250988845áa±º\ÄÖ -loss/FRt3/detection_loss‚3.94843813089224 /loss/FBt3/detection_loss‚3.0609240423549307 %loss/total_loss‚54.55905917590751 /loss/FRt1/assignment_loss‚2.825069866861616 2loss/SRqqt1/assignment_loss‚1.4738335005939007 .loss/FRt2/assignment_loss‚3.32616563921883 /loss/FRt1/detection_loss‚1.6579696829698929 /loss/FRt4/detection_loss‚2.6850157394641783 0loss/SRqqt4/detection_loss‚2.484635385774797 .loss/FBt4/detection_loss‚2.430594325065613 epoch‚2 1loss/FBt1/assignment_loss‚0.06445595250988845 /loss/FRt4/assignment_loss‚4.639316244823177 2loss/SRqqt3/assignment_loss‚2.3283715265638687 0loss/FBt3/assignment_loss‚0.2698384848263231 1loss/SRqqt2/detection_loss‚2.6541971219213387 trainer/global_step‚2549 # _timestamp‚1.78847431235035e+09 _runtime‚4519.107303257 1loss/FBt2/assignment_loss‚0.22588958367414205 /loss/FRt2/detection_loss‚2.3123105244178856 1loss/SRqqt1/detection_loss‚0.9174248453516226 /loss/FBt1/detection_loss‚2.7438236847519875 /loss/FRt3/assignment_loss‚6.104186021364653 2loss/SRqqt2/assignment_loss‚2.2540104608786735 2loss/SRqqt4/assignment_loss‚2.4775215858413326 1loss/FBt4/assignment_loss‚0.36077701678389973 1loss/SRqqt3/detection_loss‚2.4351040808593525 /loss/FBt2/detection_loss‚0.8791857281433685 _step‚209у+i zÅ‚(:ï ÓççÔãô›x disk.sda.in‚0 disk.drbd1260.out‚0.640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1487.11328125! proc.memory.rssMB‚ 3367.765625 disk.sda3.out‚ 139.2109375" disk.nvme0c0n1.in‚ 13032.765625 disk.dm-1.out‚ 139.2109375' disk.nvme0n1.out‚110706.99267578125 disk.sda1.in‚0# memory_percent‚90.9233808517456) proc.memory.percent‚20.55521011352539$ disk./.usageGB‚1161.782382965088 disk.sda.out‚ 139.2109375 disk.sda3.in‚0 cpu‚25.035008078427957 proc.cpu.threads‚17 network.sent‚ 6.380479e+06 network.recv‚ 1.473656e+06) disk./.usagePercent‚77.96819052113467) disk.nvme0c0n1.out‚110706.99267578125$ disk.dm-0.out‚110706.99267578125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 13032.765625 disk.dm-0.in‚ 13032.765625^cëÿ’Æ‚(:‡ ÓççÔù•y gpu.0.gpu‚61.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.882' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È çבÇ‚(:† âççÔ£™Áx gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.964' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™X{È‚(:ð âççÔ¦Ùõz disk.sda.out‚ 139.6171875 disk.drbd1260.out‚ 0.64453125! proc.memory.rssMB‚ 3367.765625 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 13032.765625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 13032.765625( disk.nvme0c0n1.out‚111078.2861328125 disk.dm-0.in‚ 13032.765625 disk.dm-1.in‚0 disk.dm-1.out‚ 139.6171875 cpu‚25.035834094420007 network.sent‚ 6.488906e+06 network.recv‚ 1.478056e+06% disk./.usageGB‚1161.7589149475098 disk.sda2.out‚0 disk.sda3.out‚ 139.6171875$ memory_percent‚90.92342853546143) proc.memory.availableMB‚ 1487.10546875) proc.memory.percent‚20.55521011352539) disk./.usagePercent‚77.96661556278403& disk.nvme0n1.out‚111078.2861328125 disk.sda2.in‚0# disk.dm-0.out‚111078.2861328125 disk.sda.in‚06AŸÃÉ‚(:… ñççÔ‡‡òx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.789' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3578 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iã*UÊ‚(:ô ñççÔÔ µy disk.sda1.out‚0$ memory_percent‚90.92342853546143% disk./.usageGB‚1160.3156433105469 disk.dm-1.out‚ 140.1640625" disk.nvme0n1.in‚13127.00390625 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 140.1640625 network.sent‚ 6.493221e+06) disk./.usagePercent‚77.86975639224188( disk.nvme0c0n1.out‚111198.7197265625 disk.dm-0.in‚13127.00390625 disk.drbd1260.in‚0# disk.dm-0.out‚111198.7197265625 disk.sda.out‚ 140.1640625 disk.drbd1260.out‚ 0.64453125 disk.sda2.in‚0) proc.memory.availableMB‚ 1487.10546875! proc.memory.rssMB‚ 3367.765625) proc.memory.percent‚20.55521011352539 cpu‚25.0366484814231 network.recv‚ 1.482857e+06$ disk.nvme0c0n1.in‚13127.00390625 disk.dm-1.in‚0 disk.sda.in‚0& disk.nvme0n1.out‚111198.7197265625 disk.sda1.in‚0NTOÜË‚(:„ €èçÔljçw gpu.0.gpu‚50.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚283.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úè §jÌ‚(:ß €èçÔ‹”ìx disk.drbd1260.in‚0' disk.nvme0c0n1.out‚111449.462890625 disk.sda.in‚0 disk.nvme0n1.in‚ 13178.6875 disk.sda1.in‚0$ memory_percent‚90.92345237731934( proc.memory.availableMB‚ 1487.1015625! proc.memory.rssMB‚ 3367.765625) proc.memory.percent‚20.55521011352539 network.sent‚ 6.498055e+06" disk.dm-0.out‚111449.462890625 disk.sda.out‚ 140.53125 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.037500332460155 proc.cpu.threads‚17 network.recv‚ 1.486052e+06) disk./.usagePercent‚77.86975767227955% disk./.usageGB‚1160.3156623840332 disk.drbd1260.out‚ 0.6484375% disk.nvme0n1.out‚111449.462890625 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 13178.6875 disk.dm-0.in‚ 13178.6875 disk.dm-1.in‚0 disk.dm-1.out‚ 140.53125 disk.sda3.in‚0 disk.sda3.out‚ 140.53125%—]™’Í‚(:‡ èçÔ‡ÏÄz gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚280.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øá6s΂(:è èçÔ—ôz" disk.dm-0.out‚111700.224609375 disk.dm-1.out‚ 140.95703125 network.recv‚ 1.490556e+06% disk./.usageGB‚1159.2535781860352 disk.sda3.out‚ 140.95703125 disk.drbd1260.out‚ 0.6484375% disk.nvme0n1.out‚111700.224609375 disk.nvme0c0n1.in‚ 13178.6875) proc.memory.availableMB‚ 1487.09765625! proc.memory.rssMB‚ 3367.765625 disk.nvme0n1.in‚ 13178.6875 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚90.92347621917725 cpu‚25.037624282355768 proc.cpu.threads‚17 network.sent‚ 6.506347e+06 disk.sda3.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚111700.224609375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 140.95703125) proc.memory.percent‚20.55521011352539( disk./.usagePercent‚77.7984803106039 disk.drbd1260.in‚0 disk.dm-0.in‚ 13178.6875®,¸ç2Ï- _wandbruntime‚4598  _runtime‚4598\5 ÈÄо % lr-AdamW/pg1‚0.000625955427707412 % lr-AdamW/pg2‚0.000625955427707412 trainer/global_step‚2599 $ _timestamp‚1.788474391672556e+09 _runtime‚4598.428167828 _step‚210/Å„¸ÉÑà % lr-AdamW/pg2‚0.000625955427707412 trainer/global_step‚2599 $ _timestamp‚1.788474391672556e+09 _runtime‚4598.428167828 _step‚210 % lr-AdamW/pg1‚0.000625955427707412ÒQK2Ò- _wandbruntime‚4598  _runtime‚4598–É­“(ÓúO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65141946 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6327694486299 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120215601 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65168285 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8725533623808 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419766687816192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55073655 % _timestamp‚1.7884743916736443e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58614226 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58613468 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6335199042048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403507774036304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419766687816192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 _step‚211 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419770324868096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413431515165696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55073661 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23763746 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409831859859923 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4598.429294591 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382923996915200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6338809702400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚758 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚374198514494464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34849727 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409835493950571 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413431488774144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6338809702400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23764499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚374198463291392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8725607659520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65168285 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403507799464272 trainer/global_step‚2599 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413431488774144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345ÊÁãXš$ÔÿO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8725533623808 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419766687816192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55073655 % _timestamp‚1.7884743916736443e+09 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58614226 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58613468 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6335199042048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403507774036304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419766687816192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 _step‚211 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚419770324868096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚413431515165696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55073661 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23763746 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409831859859923 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4598.429294591 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382923996915200 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6338809702400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚758 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚374198514494464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34849727 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409835493950571 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚413431488774144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6338809702400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23764499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚374198463291392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8725607659520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65168285 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403507799464272 trainer/global_step‚2599 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚413431488774144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65141946 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonÃZ~zkitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6327694486299 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120215601 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65168285 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648ÓzÎA 2Õ- _wandbruntime‚4599  _runtime‚4599A9Ï 'Ö„N ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403658128705408 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 _runtime‚4599.441235413 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6337609523712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23772621 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120286963 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419926512922112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6330100789793 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2599 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65166513 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120260592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55094085 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23773341 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58635148 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409988229495201 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34862532 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403658103277440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413585266333184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8728931324416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58635873 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚383066736207872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6337609523712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120286963 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55094079 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6326492126905 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55094085 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65192878 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6341220197376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374337753680384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34862527 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532?8'׉N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419926512922112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6341220197376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65192878 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6341220197376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374337753680384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34862527 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚403658128705408 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 _runtime‚4599.441235413 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6337609523712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23772621 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120286963 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419926512922112 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6330100789793 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2599 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65166513 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120260592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55094085 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚725 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23773341 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58635148 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409988229495201 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34862532 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚403658103277440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚413585266333184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8728931324416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58635873 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚383066736207872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6337609523712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120286963 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55094079 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6326492126905 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55094085 P×ûŽß‚(:ƒ ¼èçÔ°Î{ gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚147.45' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.49 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GæžÝ€à‚(:õ ¼èçÔ‡ØÑ} disk.dm-1.out‚ 142.26171875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚20.55521011352539 network.recv‚ 1.503998e+06$ disk.nvme0c0n1.in‚13311.30078125( disk.nvme0c0n1.out‚112327.8408203125 disk.dm-0.in‚13311.30078125 disk.drbd1260.out‚0.65625& disk.nvme0n1.out‚112327.8408203125$ memory_percent‚90.92526435852051( proc.memory.availableMB‚ 1486.8046875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 142.26171875 disk.drbd1260.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 3367.765625 cpu‚25.038293117889936 network.sent‚ 6.622937e+06) disk./.usagePercent‚77.79848133463403% disk./.usageGB‚1159.2535934448242 disk.sda.out‚ 142.26171875" disk.nvme0n1.in‚13311.30078125 proc.cpu.threads‚17# disk.dm-0.out‚112327.8408203125J¶}á‚(:ò ËèçÔͼùw disk.dm-1.in‚0 disk.dm-1.out‚ 142.6484375 disk.sda3.in‚0 disk.drbd1260.out‚0.65625" disk.nvme0n1.in‚13311.30078125 disk.sda2.in‚0! proc.memory.rssMB‚ 3367.765625) proc.memory.percent‚20.55521011352539 network.sent‚ 6.629355e+06) disk./.usagePercent‚77.79848747881486 disk.dm-0.in‚13311.30078125 disk.sda.out‚ 142.6484375( disk.nvme0c0n1.out‚112688.7822265625# disk.dm-0.out‚112688.7822265625 disk.sda.in‚0 disk.sda3.out‚ 142.6484375& disk.nvme0n1.out‚112688.7822265625 disk.sda2.out‚0( proc.memory.availableMB‚ 1487.0546875 proc.cpu.threads‚17 network.recv‚ 1.508775e+06% disk./.usageGB‚1159.2536849975586$ disk.nvme0c0n1.in‚13311.30078125 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚90.92373847961426 cpu‚25.039288109227513WF††›â‚(: ËèçÔ¸î—y gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚112.61' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.522000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ Æÿ’ã‚(:‡ ÚèçÔɘÒw gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚117.539' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÐÊB‡ä‚(:ü ÚèçÔØû®x" disk.nvme0n1.in‚13311.30078125 disk.sda2.in‚0$ disk.dm-0.out‚112954.75439453125) proc.memory.percent‚20.55521011352539 network.recv‚ 1.512297e+06) disk./.usagePercent‚77.79183742710543 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚90.92376232147217 disk.sda.in‚0$ disk.nvme0c0n1.in‚13311.30078125) disk.nvme0c0n1.out‚112954.75439453125) proc.memory.availableMB‚ 1487.05078125! proc.memory.rssMB‚ 3367.765625 cpu‚25.040209787514968 proc.cpu.threads‚17% disk./.usageGB‚1159.1545944213867 disk.dm-1.in‚0 disk.sda.out‚ 142.98828125 disk.sda3.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚112954.75439453125 disk.sda1.in‚0 disk.dm-0.in‚13311.30078125 network.sent‚ 6.636188e+06 disk.dm-1.out‚ 142.98828125 disk.sda3.out‚ 142.98828125 disk.drbd1260.out‚ 0.66015625»×Bº2å- _wandbruntime‚4673  _runtime‚4673‰Âæ¼ % _timestamp‚1.7884744664008615e+09 _runtime‚ 4673.15648443 _step‚214 $ lr-AdamW/pg1‚0.00062408623193166 $ lr-AdamW/pg2‚0.00062408623193166 trainer/global_step‚2649¦+¬ÇçÁ _runtime‚ 4673.15648443 _step‚214 $ lr-AdamW/pg1‚0.00062408623193166 $ lr-AdamW/pg2‚0.00062408623193166 trainer/global_step‚2649 % _timestamp‚1.7884744664008615e+09Ö9#Ä2è- _wandbruntime‚4673  _runtime‚4673VK«T éùO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56094861 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚750 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2649 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35489972 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427579741317120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122468503 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56094855 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24206735 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427576104265216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427576104265216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66399987 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚421120393117696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59697457 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390054878179840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8891844235776 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚745 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421120366726144 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66373648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381162982740992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56094861 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417468650672231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6455737539072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59696707 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6444414579127 _runtime‚4673.157998193 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6459348199424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122494848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35489977 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 $ _timestamp‚1.788474466402009e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417472284762879 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411024236093104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421120393117696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚421120366726144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784Cm(êþO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚421120393117696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59697457 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390054878179840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8891844235776 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚745 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421120366726144 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66373648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381162982740992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56094861 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417468650672231 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6455737539072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59696707 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6444414579127 _runtime‚4673.157998193 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6459348199424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122494848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35489977 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 $ _timestamp‚1.788474466402009e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417472284762879 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚411024236093104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421120393117696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚421120366726144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56094861 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚750 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2649 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35489972 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427579741317120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122468503 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56094855 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24206735 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427576104265216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427576104265216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66399987 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187×}g·ñ2ë- _wandbruntime‚4674  _runtime‚4674NçÙ¶ 'ì„N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56115285 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66424580 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390197377731584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6446820882413 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381302324333056 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 % _timestamp‚1.7884744674475582e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6458148020736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122539865 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚411174565334240 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411174590762208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56115285 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59718238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚4674.204384001 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122513494 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56115279 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35502782 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66398215 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427735929371136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427732292305920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421274170676736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24216170 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390197505054208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59718952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8895104601600 trainer/global_step‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122539865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 Fí‰N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421274144285184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚411174565334240 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚411174590762208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56115285 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59718238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚4674.204384001 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122513494 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56115279 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35502782 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66398215 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427735929371136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427732292305920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421274170676736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24216170 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390197505054208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59718952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8895104601600 trainer/global_step‚2649 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122539865 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_tra›zIÑin_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381302324333056 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 % _timestamp‚1.7884744674475582e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6458148020736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122539865 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FWïÑ 2loss/SRqqt2/assignment_loss‚1.5838427647948266 .loss/FRt1/detection_loss‚1.966947169793114 _step‚217 1loss/FBt2/assignment_loss‚0.17650703363679973 trainer/global_step‚2649 /loss/FRt1/assignment_loss‚2.959543997147044 0loss/SRqqt1/detection_loss‚1.128561403729722 /loss/FBt3/detection_loss‚2.3570904806256294 %loss/total_loss‚52.26298933955904 epoch‚2 _runtime‚4674.205809738 $ _timestamp‚1.788474467450253e+09 /loss/FRt3/assignment_loss‚5.336266021728516 0loss/FRt4/assignment_loss‚3.8295442376817976 2loss/SRqqt1/assignment_loss‚1.4908786396949718 2loss/FBt1/assignment_loss‚0.007922051538116648 /loss/FRt3/detection_loss‚3.2282452249526976 .loss/FRt4/detection_loss‚3.039347434043884 0loss/SRqqt3/detection_loss‚2.766908122226596 1loss/SRqqt3/assignment_loss‚2.094127011951059 1loss/SRqqt4/detection_loss‚2.1309230567320534 .loss/FBt1/detection_loss‚4.101658441126347 /loss/FBt4/detection_loss‚2.4107187603201186 0loss/FBt4/assignment_loss‚0.3245499636977911 2loss/SRqqt4/assignment_loss‚1.9511422816178072 .loss/FRt2/detection_loss‚1.843533170268391 /loss/FBt2/detection_loss‚0.9282818732508382 /loss/FBt3/assignment_loss‚0.652295867967041 1loss/SRqqt2/detection_loss‚2.9305036306381225 0loss/FRt2/assignment_loss‚3.0236507003957573÷ã×\ðÖ /loss/FRt1/assignment_loss‚2.959543997147044 0loss/SRqqt1/detection_loss‚1.128561403729722 /loss/FBt3/detection_loss‚2.3570904806256294 %loss/total_loss‚52.26298933955904 epoch‚2 _runtime‚4674.205809738 $ _timestamp‚1.788474467450253e+09 /loss/FRt3/assignment_loss‚5.336266021728516 0loss/FRt4/assignment_loss‚3.8295442376817976 2loss/SRqqt1/assignment_loss‚1.4908786396949718 2loss/FBt1/assignment_loss‚0.007922051538116648 /loss/FRt3/detection_loss‚3.2282452249526976 .loss/FRt4/detection_loss‚3.039347434043884 0loss/SRqqt3/detection_loss‚2.766908122226596 1loss/SRqqt3/assignment_loss‚2.094127011951059 1loss/SRqqt4/detection_loss‚2.1309230567320534 .loss/FBt1/detection_loss‚4.101658441126347 /loss/FBt4/detection_loss‚2.4107187603201186 0loss/FBt4/assignment_loss‚0.3245499636977911 2loss/SRqqt4/assignment_loss‚1.9511422816178072 .loss/FRt2/detection_loss‚1.843533170268391 /loss/FBt2/detection_loss‚0.9282818732508382 /loss/FBt3/assignment_loss‚0.652295867967041 1loss/SRqqt2/detection_loss‚2.9305036306381225 0loss/FRt2/assignment_loss‚3.0236507003957573 2loss/SRqqt2/assignment_loss‚1.5838427647948266 .loss/FRt1/detection_loss‚1.966947169793114 _step‚217 1loss/FBt2/assignment_loss‚0.17650703363679973 trainer/global_step‚2649Ù’0ñ‚(:’ éèçÔ¯£Ïx gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚242.081' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.416199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i³¢†ò‚(:û éèçÔ›¥¤z network.recv‚ 1.517115e+06) disk./.usagePercent‚77.79178084944037 disk.dm-1.out‚ 143.48828125 disk.sda.in‚0 disk.sda.out‚ 143.48828125' disk.nvme0n1.out‚113050.16845703125 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚113050.16845703125 disk.dm-0.in‚13311.30078125$ disk.dm-0.out‚113050.16845703125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.66015625 disk.sda1.out‚0) proc.memory.availableMB‚ 1486.49609375 cpu‚25.040324165208247$ disk./.usageGB‚1159.153751373291$ disk.nvme0c0n1.in‚13311.30078125 disk.dm-1.in‚0$ memory_percent‚90.92714786529541! proc.memory.rssMB‚ 3367.765625) proc.memory.percent‚20.55521011352539 proc.cpu.threads‚17 network.sent‚ 6.640562e+06 disk.sda3.out‚ 143.48828125 disk.drbd1260.in‚0" disk.nvme0n1.in‚13311.30078125 disk.sda1.in‚0ܧl\ƒó‚(:ø øèçÔêùx disk.sda1.out‚0$ memory_percent‚90.92421531677246 cpu‚25.04057811632068 network.sent‚ 6.748477e+06 disk.sda.out‚ 143.95703125 disk.sda3.in‚0' disk.nvme0n1.out‚113548.67724609375) proc.memory.percent‚20.55521011352539 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚13436.69140625) disk.nvme0c0n1.out‚113548.67724609375$ disk.dm-0.out‚113548.67724609375 disk.sda3.out‚ 143.95703125 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1486.9765625! proc.memory.rssMB‚ 3367.765625 network.recv‚ 1.523076e+06) disk./.usagePercent‚77.72209867071057 disk.dm-1.out‚ 143.95703125 disk.drbd1260.out‚ 0.6640625" disk.nvme0n1.in‚13436.69140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ disk./.usageGB‚1158.115436553955 disk.dm-0.in‚13436.69140625 disk.dm-1.in‚0 disk.sda.in‚0Pð@ô‚(:… øèçÔÖ±y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z¯ëÏ~õ‚(:ó ‡éçÔóøïx network.sent‚ 6.75476e+06 network.recv‚ 1.5277e+06$ disk./.usageGB‚1158.274757385254$ disk.dm-0.out‚113808.54833984375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 3367.765625) disk.nvme0c0n1.out‚113808.54833984375 disk.dm-1.in‚0 disk.dm-1.out‚ 144.3671875 disk.sda.in‚0" disk.nvme0n1.in‚13436.69140625 disk.sda2.out‚0$ memory_percent‚90.92574119567871) proc.memory.percent‚20.55521011352539$ disk.nvme0c0n1.in‚13436.69140625 disk.sda3.out‚ 144.3671875' disk.nvme0n1.out‚113808.54833984375 disk.sda2.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1486.7265625 proc.cpu.threads‚17) disk./.usagePercent‚77.73279082537812 disk.dm-0.in‚13436.69140625 disk.sda.out‚ 144.3671875 disk.drbd1260.out‚ 0.6640625 cpu‚25.041284721687326|çJ‘ö‚(:† ‡éçÔµã´y gpu.0.gpu‚53.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚150.143' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ4-·÷‚(:„ –éçÔçˆÆ~ gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝyÞ~~ø‚(:ó –éçÔ¸Ÿ network.sent‚ 6.759733e+06 network.recv‚ 1.53101e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.66796875" disk.nvme0n1.in‚13436.69140625 disk.dm-1.out‚ 144.828125) proc.memory.availableMB‚ 1487.22265625 cpu‚25.041635483801088) disk./.usagePercent‚77.73309240225346% disk./.usageGB‚1158.2792510986328& disk.nvme0n1.out‚113909.2509765625 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.out‚ 144.828125 disk.sda3.out‚ 144.828125 disk.sda2.in‚0 disk.dm-0.in‚13436.69140625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚90.92271327972412! proc.memory.rssMB‚ 3367.765625 proc.cpu.threads‚17 disk.sda1.out‚0$ disk.nvme0c0n1.in‚13436.69140625( disk.nvme0c0n1.out‚113909.2509765625# disk.dm-0.out‚113909.2509765625) proc.memory.percent‚20.55521011352539€Xù‚(:ö ¥éçÔÆÿ”x disk.sda2.out‚0( disk.nvme0c0n1.out‚114244.9345703125$ memory_percent‚90.92679023742676! proc.memory.rssMB‚ 3367.765625 network.recv‚ 1.535664e+06& disk.nvme0n1.out‚114244.9345703125( proc.memory.availableMB‚ 1486.5546875) proc.memory.percent‚20.55521011352539 cpu‚25.04222484273889 disk.sda.in‚0 disk.sda3.out‚ 145.45703125 disk.drbd1260.out‚ 0.66796875 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚13436.69140625 disk.dm-0.in‚13436.69140625# disk.dm-0.out‚114244.9345703125 network.sent‚ 6.765895e+06) disk./.usagePercent‚77.73309445031373$ disk./.usageGB‚1158.279281616211 disk.sda.out‚ 145.45703125 disk.sda3.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 145.45703125" disk.nvme0n1.in‚13436.69140625 disk.sda2.in‚0É"»#‘ú‚(:† ¥éçÔƒýÉz gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚175.239' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0478 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èîPÄ2û- _wandbruntime‚4752  _runtime‚4752ªAqÇüÁ & lr-AdamW/pg2‚0.0006221549000208112 trainer/global_step‚2699 % _timestamp‚1.7884745460227072e+09 _runtime‚4752.778289413 _step‚218 & lr-AdamW/pg1‚0.0006221549000208112Æ?$ÃÌýÆ & lr-AdamW/pg2‚0.0006221549000208112 trainer/global_step‚2699 % _timestamp‚1.7884745460227072e+09 _runtime‚4752.778289413 _step‚218 & lr-AdamW/pg1‚0.0006221549000208112Útwû2þ- _wandbruntime‚4752  _runtime‚4752›Óæ‚(ÿúO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6576276036096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6576276036096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67605350 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚418540723577872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388127553393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24648792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435385520714240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124747750 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57116061 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 trainer/global_step‚2699 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418540698149904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6568351997315 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67631689 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60779745 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚731 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6564743334635 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4752.779340424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57116055 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67631689 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚219 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397185415315968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388127502190592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425105441484539 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124721405 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6579886696448 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428809271069696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428809244678144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435389157766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435385520714240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9057861922304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67605350 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425109075575187 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428809244678144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428809271069696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚397185290077184 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67605350 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚418540723577872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388127553393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24648792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435385520714240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124747750 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57116061 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 trainer/global_step‚2699 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418540698149904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6568351997315 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67631689 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60779745 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚731 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6564743334635 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4752.779340424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57116055 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67631689 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚219 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397185415315968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388127502190592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425105441484539 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124721405 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6579886696448 JDeviceStatsMonitor.on_3]Ñò¢ train_batch_start/active.small_pool.peak‚32001 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428809271069696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428809244678144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚435389157766144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435385520714240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9057861922304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67605350 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425109075575187 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428809244678144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚428809271069696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚397185290077184 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124792767 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60801215 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36143027 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 _runtime‚4754.463777839 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428963048628736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428963022237184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428963048628736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 _step‚220 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388266792579584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60800506 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124766396 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6582297191424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6578686517760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124766396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24657479 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124792767 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428963022237184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36143032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67629917 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425261811119817 trainer/global_step‚2699 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24658183 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397327891358208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57136485 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57136479 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124792767 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428963022237184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36143032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67629917 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚425261811119817 trainer/global_step‚2699 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24658183 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397327891358208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57136485 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57136479 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124792767 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60801215 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36143027 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 _runtime‚4754.463777839 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428963048628736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428963022237184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428963048628736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 _step‚220 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388266792579584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60800506ÜåUß2„- _wandbruntime‚4754  _runtime‚4754³–U…Ï /loss/FRt1/assignment_loss‚2.460812209527704 /loss/FRt2/assignment_loss‚3.113916996363047 .loss/FRt3/detection_loss‚3.262539562971696 .loss/FBt1/detection_loss‚3.778448686003685 /loss/FBt3/detection_loss‚2.4252710714936256 _runtime‚4754.467270994 _step‚221 2loss/SRqqt2/assignment_loss‚2.1599419082586584 0loss/FBt2/assignment_loss‚0.3032449332035837 0loss/FBt4/assignment_loss‚0.4529556650055132 epoch‚2 1loss/SRqqt1/assignment_loss‚1.587675468934079 /loss/FRt4/detection_loss‚2.6118847700682553 /loss/SRqqt3/detection_loss‚2.45830948750178 % _timestamp‚1.7884745477108314e+09 %loss/total_loss‚51.89271383388012 1loss/SRqqt4/assignment_loss‚2.485401287674904 0loss/FBt1/assignment_loss‚0.4289570982873556 /loss/FRt2/detection_loss‚2.6842090282085778 1loss/SRqqt1/detection_loss‚1.1662682273735603 1loss/SRqqt4/detection_loss‚2.8116507313468238 /loss/FBt4/detection_loss‚2.2310783894438493 .loss/FRt1/detection_loss‚1.497928929357472 /loss/FRt3/assignment_loss‚4.982633528502091 /loss/FRt4/assignment_loss‚4.109844443472949 1loss/FBt3/assignment_loss‚0.35479546550050145 2loss/SRqqt3/assignment_loss‚1.4302053200701872 1loss/SRqqt2/detection_loss‚2.2954237988361945 /loss/FBt2/detection_loss‚0.7993168264740345 trainer/global_step‚2699gh"Z†Ô %loss/total_loss‚51.89271383388012 1loss/SRqqt4/assignment_loss‚2.485401287674904 0loss/FBt1/assignment_loss‚0.4289570982873556 /loss/FRt2/detection_loss‚2.6842090282085778 1loss/SRqqt1/detection_loss‚1.1662682273735603 1loss/SRqqt4/detection_loss‚2.8116507313468238 /loss/FBt4/detection_loss‚2.2310783894438493 .loss/FRt1/detection_loss‚1.497928929357472 /loss/FRt3/assignment_loss‚4.982633528502091 /loss/FRt4/assignment_loss‚4.109844443472949 1loss/FBt3/assignment_loss‚0.35479546550050145 2loss/SRqqt3/assignment_loss‚1.4302053200701872 1loss/SRqqt2/detection_loss‚2.2954237988361945 /loss/FBt2/detection_loss‚0.7993168264740345 trainer/global_step‚2699 /loss/FRt1/assignment_loss‚2.460812209527704 /loss/FRt2/assignment_loss‚3.113916996363047 .loss/FRt3/detection_loss‚3.262539562971696 .loss/FBt1/detection_loss‚3.778448686003685 /loss/FBt3/detection_loss‚2.4252710714936256 _runtime‚4754.467270994 _step‚221 2loss/SRqqt2/assignment_loss‚2.1599419082586584 0loss/FBt2/assignment_loss‚0.3032449332035837 0loss/FBt4/assignment_loss‚0.4529556650055132 epoch‚2 1loss/SRqqt1/assignment_loss‚1.587675468934079 /loss/FRt4/detection_loss‚2.6118847700682553 /loss/SRqqt3/detection_loss‚2.45830948750178 % _timestamp‚1.7884745477108314e+09Ýß);q‡‚(:æ ´éçÔ“è›y& proc.memory.availableMB‚ 1486.28125! proc.memory.rssMB‚ 3367.765625 cpu‚25.042959523522537 proc.cpu.threads‚17 network.sent‚ 6.770729e+06 network.recv‚ 1.538818e+06 disk.nvme0n1.in‚ 13561.453125 disk.sda1.out‚0) disk.nvme0c0n1.out‚114340.62255859375 disk.sda.out‚ 145.90625 disk.sda3.out‚ 145.90625 disk.drbd1260.out‚0.671875 disk.dm-0.in‚ 13561.453125 disk.dm-1.in‚0 disk.drbd1260.in‚0) proc.memory.percent‚20.55521011352539$ disk./.usageGB‚1158.342514038086 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 13561.453125$ disk.dm-0.out‚114340.62255859375 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚90.92845916748047( disk./.usagePercent‚77.7373380312013' disk.nvme0n1.out‚114340.62255859375 disk.sda1.in‚0 disk.dm-1.out‚ 145.90625Nô¬‘ˆ‚(:† ´éçÔ˜þ€z gpu.0.gpu‚34.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚167.427' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Imòɉ‚(:… ÃéçÔÈ®Êx gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚179.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gº£ƒyŠ‚(:î ÃéçÔÑÖ„y disk.dm-1.out‚ 146.3359375 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 cpu‚25.043599496106165 network.recv‚ 1.546059e+06' disk.nvme0n1.out‚114446.95068359375 disk.sda1.out‚0) proc.memory.percent‚20.55521011352539 network.sent‚ 6.87969e+06$ disk./.usageGB‚1158.381576538086) disk.nvme0c0n1.out‚114446.95068359375$ disk.dm-0.out‚114446.95068359375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0# memory_percent‚90.9254789352417) disk./.usagePercent‚77.73995954835192 disk.sda.out‚ 146.3359375 disk.sda3.out‚ 146.3359375 disk.drbd1260.out‚0.671875 disk.nvme0n1.in‚ 13561.453125) proc.memory.availableMB‚ 1486.76953125! proc.memory.rssMB‚ 3367.765625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 13561.453125 disk.dm-0.in‚ 13561.453125 disk.dm-1.in‚0I2îM‘‹‚(:† ÒéçÔ›Ûx gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚216.629' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.3258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jAà~Œ‚(:ó ÒéçÔž¢þy' disk.nvme0n1.out‚114792.81787109375 cpu‚25.044172314994004 proc.cpu.threads‚17 network.sent‚ 6.884615e+06( disk./.usagePercent‚77.7499387220366 disk.sda2.out‚0# proc.memory.rssMB‚ 3367.76953125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 13561.453125) disk.nvme0c0n1.out‚114792.81787109375$ disk.dm-0.out‚114792.81787109375 disk.sda.in‚0$ memory_percent‚90.92705249786377) proc.memory.availableMB‚ 1486.51171875( proc.memory.percent‚20.5552339553833" disk./.usageGB‚1158.5302734375 disk.dm-0.in‚ 13561.453125 disk.dm-1.in‚0 disk.dm-1.out‚ 146.89453125 disk.sda.out‚ 146.89453125 disk.sda3.out‚ 146.89453125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.67578125 network.recv‚ 1.549713e+06 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 13561.453125„Ã’’‚(:‡ áéçÔÎÔÝw gpu.0.gpu‚82.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚267.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`JS€Ž‚(:õ áéçÔèâÿx disk.sda1.out‚0 network.sent‚ 6.890772e+06) disk./.usagePercent‚77.74990288098181 disk.sda.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 13561.453125( proc.memory.availableMB‚ 1486.7578125 cpu‚25.04486435974327 proc.cpu.threads‚17 network.recv‚ 1.554429e+06% disk./.usageGB‚1158.5297393798828$ disk.dm-0.out‚114946.07958984375 disk.dm-1.out‚ 147.41015625 disk.sda3.out‚ 147.41015625 disk.drbd1260.out‚ 0.67578125 disk.nvme0n1.in‚ 13561.453125 disk.sda2.out‚0 disk.sda.out‚ 147.41015625 disk.sda3.in‚0' disk.nvme0n1.out‚114946.07958984375) disk.nvme0c0n1.out‚114946.07958984375$ memory_percent‚90.92555046081543# proc.memory.rssMB‚ 3367.76953125( proc.memory.percent‚20.5552339553833 disk.dm-0.in‚ 13561.453125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0€B.}‚(:ò ðéçԸЯy disk.sda1.out‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 13672.3671875 disk.sda1.in‚0) disk.nvme0c0n1.out‚115103.59912109375 disk.dm-1.in‚0 disk.dm-1.out‚ 147.98046875 disk.sda.in‚0 disk.sda3.out‚ 147.98046875# proc.memory.availableMB‚1486.75 network.recv‚ 1.557726e+06 disk.sda2.in‚0 disk.dm-0.in‚ 13672.3671875 disk.drbd1260.out‚ 0.6796875' disk.nvme0n1.out‚115103.59912109375$ memory_percent‚90.92559814453125# proc.memory.rssMB‚ 3367.76953125( proc.memory.percent‚20.5552339553833# disk.nvme0c0n1.in‚ 13672.3671875$ disk.dm-0.out‚115103.59912109375 disk.sda.out‚ 147.98046875 disk.sda3.in‚0 cpu‚25.045487076606456 proc.cpu.threads‚17 network.sent‚ 6.895714e+06) disk./.usagePercent‚77.75010922305441$ disk./.usageGB‚1158.532814025879 disk.sda2.out‚0²ô£‚(:‡ ðéçÔâõÂy gpu.0.gpu‚61.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚123.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.½´Ew€6552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘²È/u‘‚(:ê ÿéçÔ›ýìx( proc.memory.percent‚20.5552339553833 cpu‚25.04641988548314( disk./.usagePercent‚77.7668856527807 disk.drbd1260.in‚0' disk.nvme0n1.out‚115330.13037109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1158.7827949523926 disk.sda3.in‚0! disk.nvme0n1.in‚ 13672.3671875 disk.dm-0.in‚ 13672.3671875$ disk.dm-0.out‚115330.13037109375 disk.dm-1.in‚0 disk.dm-1.out‚148.3125( proc.memory.availableMB‚ 1486.7109375 disk.sda1.in‚0 network.recv‚ 1.562964e+06 disk.sda3.out‚148.3125 disk.drbd1260.out‚ 0.6796875) disk.nvme0c0n1.out‚115330.13037109375 disk.sda.out‚148.3125$ memory_percent‚90.92583656311035# proc.memory.rssMB‚ 3367.76953125 network.sent‚ 6.902111e+06 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 13672.3671875 \å’‚(:… ÿéçÔÕ©£y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05p Ã{“‚(:ð ŽêçÔÛ§†y disk.drbd1260.in‚0! disk.nvme0n1.in‚ 13672.3671875$ disk.nvme0n1.out‚115557.49609375 disk.sda1.out‚0 cpu‚25.04698520806577 proc.cpu.threads‚17 network.sent‚ 6.907048e+06! disk.dm-0.out‚115557.49609375 disk.dm-1.in‚0# proc.memory.rssMB‚ 3367.76953125( proc.memory.percent‚20.5552339553833) disk./.usagePercent‚77.75560647283714 disk.sda3.out‚ 148.76953125 disk.drbd1260.out‚ 0.68359375 disk.sda2.in‚0 disk.dm-1.out‚ 148.76953125 disk.sda.in‚0 disk.sda.out‚ 148.76953125) proc.memory.availableMB‚ 1486.71484375 network.recv‚ 1.566309e+06 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 13672.3671875& disk.nvme0c0n1.out‚115557.49609375 disk.dm-0.in‚ 13672.3671875$ memory_percent‚90.92581272125244% disk./.usageGB‚1158.6147270202637cHÕž”‚(:… ŽêçÔî±½y gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚164.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹m”D2•- _wandbruntime‚4851  _runtime‚4851>¯ŠwÇ–Á % _timestamp‚1.7884746443281684e+09 _runtime‚4851.083758332 _step‚222 & lr-AdamW/pg1‚0.0006201618344615412 & lr-AdamW/pg2‚0.0006201618344615412 trainer/global_step‚2749~›{Ì—Æ _step‚222 & lr-AdamW/pg1‚0.0006201618344615412 & lr-AdamW/pg2‚0.0006201618344615412 trainer/global_step‚2749 % _timestamp‚1.7884746443281684e+09 _runtime‚4851.083758332Þ ™Ø:2˜- _wandbruntime‚4851  _runtime‚4851§ È(™úO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436498122630144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6696814533120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9224095710720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436498122630144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68837052 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58137255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25091801 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6700425193472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚432742232296847 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61863000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443198574215168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436498149021696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6685072090143 _runtime‚4851.085249999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443194937163264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6688680752823 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126974307 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68863391 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36770472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6696814533120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127000652 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404316045412352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443194937163264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36770477 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚727 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚432745866387495 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443198574215168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58137261 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 trainer/global_step‚2749 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9224023772160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58137255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395092021640192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚426057160206704 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61862273 % _timestamp‚1.7884746443290057e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25092523 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68863391 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395092072843264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚223 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68837052 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6700425193472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚426057185634672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58137261 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚436498149021696 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532ï.(šÿO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9224023772160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58137255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395092021640192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚426057160206704 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61862273 % _timestamp‚1.7884746443290057e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25092523 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68863391 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395092072843264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚223 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68837052 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6700425193472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚426057185634672 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58137261 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚436498149021696 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436498122630144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6696814533120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9224095710720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436498122630144 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68837052 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58137255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25091801 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6700425193472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚432742232296847 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61863000 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443198574215168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436498149021696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6685072090143 _runtime‚4851.085249999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443194937163264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6688680752823 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126974307 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68863391 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36770472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6696814533120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127000652 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404316045412352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443194937163264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36770477 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚727 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚432745866387495 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443198574215168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58137261 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 trainer/global_step‚2749 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921ß òáF2›- _wandbruntime‚4852  _runtime‚4852s‚îç"œ„N F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58157679 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6699225014784 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127045669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25100709 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9227430317568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _runtime‚4852.189182226 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436651900189184 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443354762269184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68887984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58157685 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚432898601932125 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9227358392320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436651926580736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127019298 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25101406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚426207489447840 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6702835688448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61883986 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚426207489447840 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6702835688448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61883986 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127045669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25100709 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9227430317568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _runtime‚4852.189182226 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436651900189184 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443354762269184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68887984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58157685 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚432898601932125 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9227358392320 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436651926580736à‘kJ™2ž- _wandbruntime‚4852  _runtime‚4852,]QŸË epoch‚2 /loss/FRt4/assignment_loss‚4.562465580304464 0loss/FBt1/assignment_loss‚0.0639005838020239 0loss/FBt3/assignment_loss‚0.5677080549066886 /loss/FRt1/detection_loss‚1.6615582902835646 /loss/FBt2/detection_loss‚0.7340236129363378 .loss/FBt3/detection_loss‚2.543364005429404 /loss/FBt4/detection_loss‚3.1426059934828015 .loss/FRt1/assignment_loss‚3.04260133535831 /loss/FRt3/assignment_loss‚6.264551520347595 2loss/SRqqt4/assignment_loss‚2.2946972187076295 /loss/FBt2/assignment_loss‚0.248041216750026 1loss/SRqqt4/detection_loss‚2.6250639174665724 trainer/global_step‚2749 _step‚225 2loss/SRqqt1/assignment_loss‚1.7850132928397813 1loss/SRqqt2/assignment_loss‚2.478710366619958 0loss/SRqqt2/detection_loss‚2.508873407487516 _runtime‚4852.190208812 1loss/SRqqt1/detection_loss‚0.9883100159588407 % _timestamp‚1.7884746454347172e+09 1loss/SRqqt3/assignment_loss‚2.991276911497116 0loss/FBt4/assignment_loss‚0.3039158229818592 .loss/FRt4/detection_loss‚3.111418056488037 0loss/SRqqt3/detection_loss‚3.044620099067688 .loss/FBt1/detection_loss‚3.964367836713791 %loss/total_loss‚58.13739889358825 .loss/FRt3/detection_loss‚3.849593862891197 /loss/FRt2/assignment_loss‚3.140060290142342 /loss/FRt2/detection_loss‚2.2206576011247106xqÇæV Ð .loss/FRt3/detection_loss‚3.849593862891197 /loss/FRt2/assignment_loss‚3.140060290142342 /loss/FRt2/detection_loss‚2.2206576011247106 epoch‚2 /loss/FRt4/assignment_loss‚4.562465580304464 0loss/FBt1/assignment_loss‚0.0639005838020239 0loss/FBt3/assignment_loss‚0.5677080549066886 /loss/FRt1/detection_loss‚1.6615582902835646 /loss/FBt2/detection_loss‚0.7340236129363378 .loss/FBt3/detection_loss‚2.543364005429404 /loss/FBt4/detection_loss‚3.1426059934828015 .loss/FRt1/assignment_loss‚3.04260133535831 /loss/FRt3/assignment_loss‚6.264551520347595 2loss/SRqqt4/assignment_loss‚2.2946972187076295 /loss/FBt2/assignment_loss‚0.248041216750026 1loss/SRqqt4/detection_loss‚2.6250639174665724 trainer/global_step‚2749 _step‚225 2loss/SRqqt1/assignment_loss‚1.7850132928397813 1loss/SRqqt2/assignment_loss‚2.478710366619958 0loss/SRqqt2/detection_loss‚2.508873407487516 _runtime‚4852.190208812 1loss/SRqqt1/detection_loss‚0.9883100159588407 % _timestamp‚1.7884746454347172e+09 1loss/SRqqt3/assignment_loss‚2.991276911497116 0loss/FBt4/assignment_loss‚0.3039158229818592 .loss/FRt4/detection_loss‚3.111418056488037 0loss/SRqqt3/detection_loss‚3.044620099067688 .loss/FBt1/detection_loss‚3.964367836713791 %loss/total_loss‚58.13739889358825á>m&}u¡‚(:ê êçÔú†¿x! disk.nvme0n1.in‚ 13672.3671875" disk.nvme0n1.out‚ 115837.546875) proc.memory.availableMB‚ 1486.41796875 network.recv‚ 1.57049e+06% disk./.usageGB‚1158.7385177612305 disk.sda1.in‚0 disk.dm-0.out‚ 115837.546875 disk.dm-1.in‚0 disk.dm-1.out‚ 149.10546875 disk.sda.in‚0 disk.sda3.out‚ 149.10546875) disk./.usagePercent‚77.76391417333079 disk.sda1.out‚0 disk.dm-0.in‚ 13672.3671875 disk.sda3.in‚0 disk.drbd1260.in‚0$ memory_percent‚90.92762470245361# proc.memory.rssMB‚ 3367.76953125( proc.memory.percent‚20.5552339553833 network.sent‚ 6.913031e+06$ disk.nvme0c0n1.out‚ 115837.546875 cpu‚25.047655785907697 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 13672.3671875 disk.sda.out‚ 149.10546875 disk.drbd1260.out‚ 0.68359375–1µÖ¢‚(:… êçÔ‰Ïy gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.882' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9764 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*å°y£‚(:î ¬êçԉˑx disk.dm-0.in‚ 13860.515625 disk.dm-1.out‚ 149.6171875' proc.memory.availableMB‚ 1486.609375# proc.memory.rssMB‚ 3367.76953125% disk./.usageGB‚1158.8388710021973 disk.sda1.out‚0 disk.sda.in‚0$ memory_percent‚90.92645645141602 cpu‚25.048219976919032 network.sent‚ 7.019734e+06 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 13860.515625$ disk.dm-0.out‚116087.40283203125 disk.dm-1.in‚0 disk.sda.out‚ 149.6171875 network.recv‚ 1.575913e+06) disk./.usagePercent‚77.77064896353404 disk.nvme0n1.in‚ 13860.515625' disk.nvme0n1.out‚116208.27783203125 disk.sda2.out‚0) disk.nvme0c0n1.out‚116208.27783203125( proc.memory.percent‚20.5552339553833 proc.cpu.threads‚17 disk.sda3.out‚ 149.6171875 disk.drbd1260.out‚0.6875 disk.sda1.in‚0Îubm¤‚(:‚ ¬êçÔ‚¼íx gpu.0.gpu‚13.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.4' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.68 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ:çz¥‚(:ï »êçÔ˦§y$ disk.dm-0.out‚116463.57470703125 disk.sda.out‚ 149.91796875 disk.sda3.in‚0$ memory_percent‚90.92645645141602' proc.memory.availableMB‚ 1486.609375 cpu‚25.04881501418992 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 13860.515625 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 7.02806e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚0.6875 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚116463.57470703125 disk.dm-1.out‚ 149.91796875 disk.sda3.out‚ 149.91796875 network.recv‚ 1.580997e+06) disk./.usagePercent‚77.80262481657573' disk.nvme0n1.out‚116463.57470703125 disk.sda2.out‚0 disk.dm-0.in‚ 13860.515625 disk.dm-1.in‚0# proc.memory.rssMB‚ 3367.76953125( proc.memory.percent‚20.5552339553833% disk./.usageGB‚1159.3153343200684 disk.nvme0n1.in‚ 13860.515625tŠ©o¦‚(:’ »êçÔ·•žy gpu.0.gpu‚59.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚218.026' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.605199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%a×Ý’§‚(:‡ ÊêçÔüôŽx gpu.0.gpu‚49.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ër¨‚(:ç ÊêçÔ¨¾úy$ memory_percent‚90.92648029327393& disk.nvme0n1.out‚116464.6826171875# disk.dm-0.out‚116464.6826171875 disk.dm-1.out‚150.3125) proc.memory.availableMB‚ 1486.60546875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 13860.515625 disk.sda1.in‚0( disk.nvme0c0n1.out‚116464.6826171875 disk.dm-0.in‚ 13860.515625# proc.memory.rssMB‚ 3367.76953125( proc.memory.percent‚20.5552339553833 cpu‚25.04941137698495 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 13860.515625 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚150.3125 disk.drbd1260.in‚0 network.sent‚ 7.030982e+06 network.recv‚ 1.584168e+06) disk./.usagePercent‚77.80262481657573% disk./.usageGB‚1159.3153343200684 disk.dm-1.in‚0 disk.sda.out‚150.3125 disk.drbd1260.out‚ 0.69140625M穾‘©‚(:† ÙêçÔÞÒïw gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚238.335' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ ÷x|ª‚(:ñ ÙêçԶѧy disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 150.6953125$ memory_percent‚90.92645645141602 network.sent‚ 7.039061e+06 disk.nvme0n1.in‚ 13860.515625 disk.sda2.out‚0) disk.nvme0c0n1.out‚116618.03857421875 disk.sda3.in‚0' proc.memory.availableMB‚ 1486.609375( proc.memory.percent‚20.5552339553833 cpu‚25.050067639318605 disk.sda1.out‚0 network.recv‚ 1.58872e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.69140625$ disk.dm-0.out‚116618.03857421875 disk.dm-1.in‚0 disk.sda.out‚ 150.6953125# proc.memory.rssMB‚ 3367.76953125 disk.dm-0.in‚ 13860.515625 disk.sda.in‚0 disk.sda3.out‚ 150.6953125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 13860.515625) disk./.usagePercent‚77.79475002482249% disk./.usageGB‚1159.1979942321777' disk.nvme0n1.out‚116618.03857421875”¾ £ƒ«‚(:ø èêçÔÃÆîw# proc.memory.rssMB‚ 3367.76953125 network.sent‚ 7.041983e+06) disk.nvme0c0n1.out‚116964.49169921875 disk.dm-1.in‚0' disk.nvme0n1.out‚116964.49169921875( proc.memory.percent‚20.5552339553833 proc.cpu.threads‚17) disk./.usagePercent‚77.80316115235996 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 13935.7421875 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1486.6015625 cpu‚25.050678781428235# disk.nvme0c0n1.in‚ 13935.7421875 disk.dm-0.in‚ 13935.7421875$ disk.dm-0.out‚116964.49169921875 disk.sda.in‚0 disk.sda.out‚ 151.30859375 disk.sda3.in‚0 disk.sda2.in‚0 network.recv‚ 1.592211e+06% disk./.usageGB‚1159.3233261108398 disk.dm-1.out‚ 151.30859375 disk.sda3.out‚ 151.30859375 disk.drbd1260.out‚ 0.6953125 disk.sda2.out‚0$ memory_percent‚90.92650413513184ŠK5p‘¬‚(:† èêçÔ“­Îy gpu.0.gpu‚58.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚223.875' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÂI12­- _wandbruntime‚4942  _runtime‚4942$ÏG†Ç®Á & lr-AdamW/pg1‚0.0006181074506057254 & lr-AdamW/pg2‚0.0006181074506057254 trainer/global_step‚2799 % _timestamp‚1.7884747355275702e+09 _runtime‚4942.283132008 _step‚226p‰’{Ì¯Æ & lr-AdamW/pg1‚0.0006181074506057254 & lr-AdamW/pg2‚0.0006181074506057254 trainer/global_step‚2799 % _timestamp‚1.7884747355275702e+09 _runtime‚4942.283132008 _step‚226âôêº×2°- _wandbruntime‚4942  _runtime‚4942Ö©õÿ'±ùO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451004353612288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚444187000582144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59158455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2799 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129253554 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚411446447832576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚444187000582144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6809009508331 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433573622263504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37410727 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59158461 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62947227 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411446570974208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451004353612288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70068754 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25536505 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938560 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 _step‚227 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59158455 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚444187026973696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402056541089792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚440382657199803 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6805400845651 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70068754 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62947936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚402056592292864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25537209 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9389906742784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚440379023109155 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6820963690496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70095093 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090öѨ²þO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚440382657199803 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6805400845651 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMoâá¿&í&nitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70068754 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62947936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚402056592292864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25537209 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9389906742784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚440379023109155 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6820963690496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70095093 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451004353612288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚444187000582144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59158455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2799 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129253554 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚411446447832576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚444187000582144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6809009508331 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433573622263504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37410727 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59158461 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62947227 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411446570974208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451004353612288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70068754 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25536505 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938560 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 _step‚227 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59158455 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚444187026973696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402056541089792ã`¥cð2³- _wandbruntime‚4944  _runtime‚4944ªiS '´„N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37423532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚411589189976064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚402195831478784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433723976932608 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚444340778141184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59178879 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚716 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59178885 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411589066847744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6807807148937 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62968857 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9393235368960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6823374185472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819763511808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70093321 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62969573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2799 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129298571 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9393307294208 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25545330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚440535392744433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129298571 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451160541652992 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6811415811825 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70119686 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚444340804532736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 % _timestamp‚1.7884747373584604e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451164178718208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129272200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25546041 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4944.116072658 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚444340804532736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129272200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6823374185472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6823374185472 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59178879 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚716 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59178885 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411589066847744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6807807148937 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62968857 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9393235368960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6823374185472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819763511808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70093321 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62969573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2799 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129298571 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9393307294208 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128320 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25545330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚440535392744433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129298571 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451160541652992 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6811415811825 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70119686 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚444340804532736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 % _timestamp‚1.7884747373584604e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451164178718208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129272200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25546041 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚4944.116072658 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚444340804532736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129272200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1äü<È/2¶- _wandbruntime‚4944  _runtime‚4944˜à7UZ·Ô 1loss/SRqqt1/detection_loss‚0.9922715926195605 /loss/FBt3/detection_loss‚2.3909244817845963 _runtime‚4944.120189956 .loss/FBt1/detection_loss‚3.791958292325338 0loss/SRqqt3/detection_loss‚2.667413926124573 $loss/total_loss‚51.2619683761651 trainer/global_step‚2799 _step‚229 2loss/FBt1/assignment_loss‚0.019084005917648028 0loss/FBt2/assignment_loss‚0.3597520842040683 0loss/SRqqt2/detection_loss‚2.911452630162239 epoch‚2 0loss/FRt2/assignment_loss‚3.0673936022364576 2loss/SRqqt4/assignment_loss‚1.9795787994717728 0loss/FBt3/assignment_loss‚0.5856226405457539 /loss/FBt2/detection_loss‚0.6271434898765104 /loss/FRt3/assignment_loss‚5.466293662786484 1loss/FBt4/assignment_loss‚0.34867636474186786 1loss/SRqqt4/detection_loss‚2.8062330011663765 % _timestamp‚1.7884747373634937e+09 0loss/FRt1/assignment_loss‚2.1678797149658204 0loss/SRqqt1/assignment_loss‚1.35812311597421 2loss/SRqqt3/assignment_loss‚1.9005917339771985 /loss/FRt2/detection_loss‚2.1352703331605247 /loss/FRt4/detection_loss‚2.9328953008497916 /loss/FBt4/detection_loss‚2.2161818146705627 0loss/FRt4/assignment_loss‚3.4082534755429914 1loss/SRqqt2/assignment_loss‚2.322824430838227 /loss/FRt1/detection_loss‚1.5258209826005622 /loss/FRt3/detection_loss‚3.2803288996219635¸°AQ ¸Ù 1loss/SRqqt1/detection_loss‚0.9922715926195605 /loss/FBt3/detection_loss‚2.3909244817845963 _runtime‚4944.120189956 .loss/FBt1/detection_loss‚3.791958292325338 0loss/SRqqt3/detection_loss‚2.667413926124573 $loss/total_loss‚51.2619683761651 trainer/global_step‚2799 _step‚229 2loss/FBt1/assignment_loss‚0.019084005917648028 0loss/FBt2/assignment_loss‚0.3597520842040683 0loss/SRqqt2/detection_loss‚2.911452630162239 epoch‚2 0loss/FRt2/assignment_loss‚3.0673936022364576 2loss/SRqqt4/assignment_loss‚1.9795787994717728 0loss/FBt3/assignment_loss‚0.5856226405457539 /loss/FBt2/detection_loss‚0.6271434898765104 /loss/FRt3/assignment_loss‚5.466293662786484 1loss/FBt4/assignment_loss‚0.34867636474186786 1loss/SRqqt4/detection_loss‚2.8062330011663765 % _timestamp‚1.7884747373634937e+09 0loss/FRt1/assignment_loss‚2.1678797149658204 0loss/SRqqt1/assignment_loss‚1.35812311597421 2loss/SRqqt3/assignment_loss‚1.9005917339771985 /loss/FRt2/detection_loss‚2.1352703331605247 /loss/FRt4/detection_loss‚2.9328953008497916 /loss/FBt4/detection_loss‚2.2161818146705627 0loss/FRt4/assignment_loss‚3.4082534755429914 1loss/SRqqt2/assignment_loss‚2.322824430838227 /loss/FRt1/detec—ܹTtion_loss‚1.5258209826005622 /loss/FRt3/detection_loss‚3.2803288996219635å=ª)w¹‚(:ö ÷êçÔû¡¤x$ disk.nvme0c0n1.in‚13966.08203125 disk.dm-0.in‚13966.08203125 disk.sda.in‚0 disk.sda.out‚ 151.7578125$ memory_percent‚90.92690944671631 network.recv‚ 1.59717e+06$ disk./.usageGB‚1159.323257446289' disk.nvme0n1.out‚117294.14794921875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 151.7578125) proc.memory.availableMB‚ 1486.53515625( proc.memory.percent‚20.5552339553833) disk./.usagePercent‚77.80315654422434 disk.sda3.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚117294.14794921875 network.sent‚ 7.048415e+06 disk.sda3.out‚ 151.7578125 disk.drbd1260.out‚ 0.6953125 disk.sda2.in‚0$ disk.dm-0.out‚117294.14794921875# proc.memory.rssMB‚ 3367.76953125 cpu‚25.05131663264744 proc.cpu.threads‚17 disk.drbd1260.in‚0" disk.nvme0n1.in‚13966.08203125 disk.sda2.out‚0¥æÙª‘º‚(:† ÷êçÔ¤Œ¤y gpu.0.gpu‚47.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹­Ù†‘»‚(:† †ëçÔ³üæw gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚195.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.1514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[={¼‚(:ð †ëçÔ½™Á| network.recv‚ 1.602521e+06$ disk./.usageGB‚1158.912696838379" disk.nvme0n1.in‚13966.08203125 disk.sda2.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3367.76953125 cpu‚25.05191111253172 proc.cpu.threads‚17) disk./.usagePercent‚77.77560347734419 disk.sda3.in‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚117463.115234375" disk.dm-0.out‚117463.115234375& proc.memory.availableMB‚ 1486.28125 network.sent‚ 7.157773e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 152.08203125 disk.drbd1260.out‚0.703125$ disk.nvme0c0n1.in‚13966.08203125 disk.dm-0.in‚13966.08203125 disk.sda.in‚0 disk.sda.out‚ 152.08203125 disk.sda3.out‚ 152.08203125 disk.sda2.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚117463.115234375$ memory_percent‚90.92845916748047( proc.memory.percent‚20.5552339553833‚Ñ«œ½‚(:‘ •ëçÔëìÅw gpu.0.gpu‚60.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.421' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.084199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôŒEW¾‚(:ô •ëçÔߎ²y disk.dm-0.in‚13966.08203125 disk.dm-1.out‚ 152.48046875 disk.drbd1260.in‚0" disk.nvme0n1.in‚13966.08203125 disk.sda1.in‚0# proc.memory.rssMB‚ 3367.76953125 network.sent‚ 7.162123e+06" disk.dm-0.out‚117808.623046875 disk.sda3.out‚ 152.48046875 disk.drbd1260.out‚0.703125% disk.nvme0n1.out‚117808.623046875$ memory_percent‚90.92695713043213( proc.memory.percent‚20.5552339553833 cpu‚25.052479387699446 network.recv‚ 1.607145e+06 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1486.52734375$ disk./.usageGB‚1158.912696838379$ disk.nvme0c0n1.in‚13966.08203125 disk.sda.in‚0 disk.sda.out‚ 152.48046875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚77.77560347734419' disk.nvme0c0n1.out‚117808.623046875ÆTàð‘¿‚(:† ¤ëçÔ°Ó‚y gpu.0.gpu‚11.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.309' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4618 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03 IzÀ‚(:ï ¤ëçÔ䢫y$ memory_percent‚90.92698097229004 network.recv‚ 1.610334e+06$ disk./.usageGB‚1159.037654876709 disk.sda.out‚ 153.23046875 disk.drbd1260.out‚ 0.70703125$ disk.nvme0c0n1.in‚13966.08203125 cpu‚25.053209211364898 disk.sda3.out‚ 153.23046875" disk.nvme0n1.in‚13966.08203125 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-0.out‚118227.5859375" proc.memory.rssMB‚ 3367.7734375) proc.memory.percent‚20.55525779724121) disk./.usagePercent‚77.78398951614331 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚13966.08203125( proc.memory.availableMB‚ 1486.5234375 proc.cpu.threads‚17 network.sent‚ 7.168887e+06# disk.nvme0n1.out‚118227.5859375 disk.sda1.in‚0% disk.nvme0c0n1.out‚118227.5859375 disk.dm-1.in‚0 disk.dm-1.out‚ 153.23046875@ÎSÁ‘Á‚(:† ³ëçÔÖïÅw gpu.0.gpu‚79.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.027 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R@À~‚(:ó ³ëçÔæê¶x disk.sda3.in‚0 disk.sda3.out‚ 153.60546875) disk./.usagePercent‚77.79241165200474% disk./.usageGB‚1159.1631507873535$ disk.nvme0c0n1.in‚14095.24609375& disk.nvme0c0n1.out‚118380.81640625 disk.dm-1.in‚0 disk.dm-1.out‚ 153.60546875 network.sent‚ 7.173508e+06 disk.dm-0.in‚14095.24609375 disk.sda.out‚ 153.60546875 disk.drbd1260.out‚ 0.70703125" disk.nvme0n1.in‚14095.24609375 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚90.92700481414795 network.recv‚ 1.615366e+06$ disk.nvme0n1.out‚118380.81640625 disk.sda1.out‚0) proc.memory.percent‚20.55525779724121 cpu‚25.05379645933205 proc.cpu.threads‚17! disk.dm-0.out‚118380.81640625 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1486.51953125" proc.memory.rssMB‚ 3367.7734375…U€ß€Ã‚(:õ ÂëçԮʄx$ disk.dm-0.out‚118655.75048828125 disk.sda3.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚118655.75048828125 disk.sda2.in‚0 disk.sda.out‚ 153.953125$ memory_percent‚90.92700481414795) proc.memory.availableMB‚ 1486.51953125% disk./.usageGB‚1159.1625785827637 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚14095.24609375 disk.dm-1.in‚0 disk.dm-1.out‚ 153.953125 disk.sda.in‚0 disk.sda3.out‚ 153.953125 network.sent‚ 7.178341e+06 network.recv‚ 1.618754e+06" disk.nvme0n1.in‚14095.24609375 disk.drbd1260.out‚ 0.7109375" proc.memory.rssMB‚ 3367.7734375) proc.memory.percent‚20.55525779724121 cpu‚25.054521740407278 proc.cpu.threads‚17( disk./.usagePercent‚77.7923732508746 disk.sda2.out‚0) disk.nvme0c0n1.out‚118655.75048828125 disk.dm-0.in‚14095.24609375£™ÈœÄ‚(:‘ ÂëçÔÃÞõy gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚140.754' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.150799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì¿t2Å- _wandbruntime‚5039  _runtime‚5039QР7ÇÆÁ & lr-AdamW/pg2‚0.0006159921765838799 trainer/global_step‚2849 % _timestamp‚1.7884748324613514e+09 _runtime‚5039.218300431 _step‚230 & lr-AdamW/pg1‚0.0006159921765838799¹jpÌÇÆ & lr-AdamW/pg1‚0.0006159921765838799 & lr-AdamW/pg2‚0.0006159921765838799 trainer/global_step‚2849 % _timestamp‚1.7884748324613514e+09 _runtime‚5039.218300431 _step‚230æŠ8ó 2È- _wandbruntime‚5039  _runtime‚5039jrEB(ÉúO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409021111742464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418577467939840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418577593178624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441090109748272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38050977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 % _timestamp‚1.7884748324651468e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71300456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451875904925696 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6941502187520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6929338263839 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6925729601159 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71300456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71326795 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25980107 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64031084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38050972 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131480111 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458817407113216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458813770061312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6937891527168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448015813921463 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25979379 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60179661 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9556407400448 _step‚231 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71326795 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441090084320304 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451875878534144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60179655 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458813770061312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451875904925696 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131506456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9556481436160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6937891527168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60179661 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448019448012111 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532ílÌÖ(ÊÿO O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451875878534144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60179655 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458813770061312 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451875904925696 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131506456 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9556481436160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6937891527168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60179661 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448019448012111 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409021111742464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418577467939840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418577593178624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441090109748272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38050977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 % _timestamp‚1.7884748324651468e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71300456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451875904925696 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6941502187520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6929338263839 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6925729601159 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71300456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71326795 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25980107 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64031084 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38050972 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131480111 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458817407113216 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458813770061312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6937891527168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448015813921463 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25979379 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60179661 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9556407400448 _step‚231 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71326795 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441090084320304çæ.rÿ…Ë‚(:ú ÑëçÔ…“÷y" proc.memory.rssMB‚ 3367.7734375) proc.memory.percent‚20.55525779724121 disk.sda1.out‚0$ disk.nvme0c0n1.in‚14095.24609375) disk.nvme0c0n1.out‚118930.25439453125 disk.dm-0.in‚14095.24609375 disk.dm-1.in‚0 cpu‚25.05517119342123 proc.cpu.threads‚17 network.recv‚ 1.623397e+06) disk./.usagePercent‚77.80920062610022% disk./.usageGB‚1159.4133186340332" disk.nvme0n1.in‚14095.24609375' disk.nvme0n1.out‚118930.25439453125 disk.sda2.in‚0 disk.sda.out‚ 154.64453125 disk.drbd1260.out‚ 0.7109375 network.sent‚ 7.184462e+06$ disk.dm-0.out‚118930.25439453125 disk.dm-1.out‚ 154.64453125 disk.sda3.in‚0 disk.sda3.out‚ 154.64453125 disk.drbd1260.in‚0$ memory_percent‚90.93108177185059( proc.memory.availableMB‚ 1485.8515625 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.in‚0܆ó|‘Ì‚(:† ÑëçÔÌÇûz gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚169.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ïþ`2Í- _wandbruntime‚5041  _runtime‚5041;¹RX ΄N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60200085 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚5041.717676341 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚452029682484736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25988210 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6931744567333 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131551473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64052689 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448172183556741 % _timestamp‚1.7884748349562197e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131525102 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6940302008832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64051987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409160350928384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9559790355968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60200079 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6943912682496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38063777 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6940302008832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38063782 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458973595167232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71351388 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6928135904445 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131551473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458973595167232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441240438989408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60200085 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚452029656093184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60200079 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409160402131456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9559716333568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚5041.717676341 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚452029682484736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25988210 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6931744567333 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131551473 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64052689 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448172183556741 % _timestamp‚1.7884748349562197e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131525102 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6940302008832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64051987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409160350928384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9559790355968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60200079 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6943912682496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38063777 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6940302008832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38063782 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458973595167232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71351388 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6928135904445 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131551473 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458973595167232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441240438989408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60200085 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚452029656093184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60200079 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409160402131456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9559716333568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9722538949120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459564756486144 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72558497 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425708044902400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚448606571805072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61200861 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459564782877696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚466626823562240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7058430024192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _runtime‚5139.312790307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61200855 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425708170141184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26421932 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72532158 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65113159 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚737 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466623186510336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7058430024192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133759358 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚455652604733771 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61200855 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7046058356667 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 _step‚235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72558497 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7049667019347 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448606546377104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9722464913408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415985631192064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚466623186510336 trainer/global_step‚2899 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133733013 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38691227 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚455656238824419 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187}E(äÿO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415985631192064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚466623186510336 trainer/global_step‚2899 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133733013 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38691227 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚455656238824419 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466626823562240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7062040684544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38691222 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9722538949120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚459564756486144 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72558497 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425708044902400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚448606571805072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61200861 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459564782877696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚466626823562240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7058430024192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _runtime‚5139.312790307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61200855 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425708170141184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26421932 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72532158 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65113159 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚737 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466623186510336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7058430024192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133759358 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚455652604733771 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61200855 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7046058356667 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 _step‚235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72558497 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7049667019347 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448606546377104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9722464913408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532ë[2ý=2å- _wandbruntime‚5141  _runtime‚5141ȧ 'æ„N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7064451179520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804375 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚466779374551040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7060840505856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65134013 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61221279 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72556725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9725790167552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416124921581056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416124870377984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455805340278193 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _step‚236 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61221285 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459718560436736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚466783011616256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38704032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72583090 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459718534045184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7064451179520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7052073322841 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7048464659953 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133778004 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61221279 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72556725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65134718 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9725864189952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459718560436736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 trainer/global_step‚2899 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459718534045184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚455808974369049 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61221285 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚705 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425850660545536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26429986 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448756901046208€ˆ˜¶!ç‰N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459718534045184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7064451179520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7052073322841 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7048464659953 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133778004 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61221279 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72556725 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65134718 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9725864189952 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459718560436736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 trainer/global_step‚2899 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459718534045184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚455808974369049 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61221285 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚705 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425850660545536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26429986 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚448756901046208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7064451179520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133804375 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚466779374551040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7060840505856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65134013 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61221279 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72556725 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9725790167552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416124921581056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416124870377984 S?DeviceStatsMonitor.onlúÍî_train_batch_end/requested_bytes.all.freed‚455805340278193 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _step‚236 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61221285 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459718560436736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚466783011616256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38704032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72583090 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448ìÙƱC2è- _wandbruntime‚5141  _runtime‚5141+»jçTéÎ 1loss/SRqqt1/assignment_loss‚1.606691211371893 .loss/FRt1/detection_loss‚1.688934100285563 1loss/SRqqt3/detection_loss‚3.0492754563689233 2loss/SRqqt3/assignment_loss‚2.0199165023863315 0loss/FBt3/assignment_loss‚0.6089901086961618 -loss/FRt3/detection_loss‚3.33080590289572 /loss/FRt4/detection_loss‚2.9303319712479907 .loss/FBt3/detection_loss‚1.950204533835252 /loss/FRt3/assignment_loss‚5.532099485397339 0loss/SRqqt4/assignment_loss‚2.75786420341694 1loss/FBt1/assignment_loss‚0.02169974723365158 .loss/FBt4/detection_loss‚2.560768874628203 trainer/global_step‚2899 % _timestamp‚1.7884749346869955e+09 _step‚237 2loss/SRqqt2/assignment_loss‚2.5931736928495495 1loss/SRqqt1/detection_loss‚1.0067325595525964 _runtime‚5141.442359627 /loss/FRt4/assignment_loss‚4.371849830945333 0loss/FBt4/assignment_loss‚0.3177438676357046 /loss/FBt1/detection_loss‚2.5703355431556703 epoch‚2 1loss/FBt2/assignment_loss‚0.33115965345640663 0loss/SRqqt4/detection_loss‚2.076452840458263 /loss/FRt1/assignment_loss‚2.988049313227336 0loss/FRt2/assignment_loss‚2.5145780459830633 /loss/FRt2/detection_loss‚2.1456175332790925 /loss/SRqqt2/detection_loss‚2.86020009896972 /loss/FBt2/detection_loss‚0.7008377999688188 &loss/total_loss‚52.534312877245526²4½ YêÓ /loss/FRt4/assignment_loss‚4.371849830945333 0loss/FBt4/assignment_loss‚0.3177438676357046 /loss/FBt1/detection_loss‚2.5703355431556703 epoch‚2 1loss/FBt2/assignment_loss‚0.33115965345640663 0loss/SRqqt4/detection_loss‚2.076452840458263 /loss/FRt1/assignment_loss‚2.988049313227336 0loss/FRt2/assignment_loss‚2.5145780459830633 /loss/FRt2/detection_loss‚2.1456175332790925 /loss/SRqqt2/detection_loss‚2.86020009896972 /loss/FBt2/detection_loss‚0.7008377999688188 &loss/total_loss‚52.534312877245526 1loss/SRqqt1/assignment_loss‚1.606691211371893 .loss/FRt1/detection_loss‚1.688934100285563 1loss/SRqqt3/detection_loss‚3.0492754563689233 2loss/SRqqt3/assignment_loss‚2.0199165023863315 0loss/FBt3/assignment_loss‚0.6089901086961618 -loss/FRt3/detection_loss‚3.33080590289572 /loss/FRt4/detection_loss‚2.9303319712479907 .loss/FBt3/detection_loss‚1.950204533835252 /loss/FRt3/assignment_loss‚5.532099485397339 0loss/SRqqt4/assignment_loss‚2.75786420341694 1loss/FBt1/assignment_loss‚0.02169974723365158 .loss/FBt4/detection_loss‚2.560768874628203 trainer/global_step‚2899 % _timestamp‚1.7884749346869955e+09 _step‚237 2loss/SRqqt2/assignment_loss‚2.5931736928495495 1loss/SRqqt1/detection_loss‚1.0067325595525964 _runtime‚5141.442359627í;Þž}ë‚(:ò ºìçÔî±åx) disk./.usagePercent‚77.84111913345423# disk.dm-0.out‚121360.4482421875 disk.dm-1.out‚ 158.0234375 disk.sda.in‚0 disk.nvme0n1.in‚ 14357.328125) proc.memory.availableMB‚ 1755.16796875# proc.memory.rssMB‚ 3090.64453125* proc.memory.percent‚18.863797187805176 network.sent‚ 7.406216e+06% disk./.usageGB‚1159.8889274597168( disk.nvme0c0n1.out‚121360.4482421875 disk.dm-0.in‚ 14357.328125 disk.dm-1.in‚0 disk.sda.out‚ 158.0234375 disk.sda3.out‚ 158.0234375 disk.drbd1260.in‚0" disk.nvme0c0n1.in‚ 14357.328125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.7265625& disk.nvme0n1.out‚121360.4482421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚89.28730487823486 network.recv‚ 1.654543e+06 disk.sda1.out‚0 cpu‚25.059562270998295 proc.cpu.threads‚17‰Ý…ä‘ì‚(:† ºìçÔ¦£²y gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚197.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.5736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×sm$œí‚(:‘ ÉìçÔªŒºw gpu.0.gpu‚42.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚164.831' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚32.96619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸V|î‚(:ñ ÉìçÔ£ÒÅx( disk.nvme0c0n1.out‚121681.9052734375 disk.dm-0.in‚ 14357.328125 disk.dm-1.in‚0) proc.memory.percent‚18.89512538909912 cpu‚25.060250858755616 network.sent‚ 7.514799e+06$ disk./.usageGB‚1159.912395477295 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚121681.9052734375 disk.sda.in‚0 disk.sda.out‚ 158.69140625 network.recv‚ 1.660677e+06 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 14357.328125 disk.dm-1.out‚ 158.69140625 disk.sda3.in‚0 disk.sda3.out‚ 158.69140625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 14357.328125) disk./.usagePercent‚77.84269409180487 disk.drbd1260.out‚ 0.7265625& disk.nvme0n1.out‚121681.9052734375$ memory_percent‚89.31875228881836' proc.memory.availableMB‚ 1750.015625# proc.memory.rssMB‚ 3095.77734375 proc.cpu.threads‚17¹s’ï‚(:‡ ØìçÔÚç†z gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚266.336' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡|ð‚(:ñ ØìçÔ£ÿ™z disk.dm-1.out‚ 159.0703125 disk.sda.out‚ 159.0703125$ memory_percent‚89.34814929962158) proc.memory.availableMB‚ 1745.19921875 proc.memory.rssMB‚ 3100.90625 cpu‚25.06082212029596 network.recv‚ 1.663647e+06% disk./.usageGB‚1159.9124183654785 disk.nvme0n1.in‚ 14357.328125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 14357.328125$ disk.dm-0.out‚122028.73974609375 disk.sda.in‚0 disk.sda3.out‚ 159.0703125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7265625* proc.memory.percent‚18.926429748535156 proc.cpu.threads‚17 network.sent‚ 7.518988e+06' disk.nvme0n1.out‚122028.73974609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚122028.73974609375 disk.dm-1.in‚0 disk.sda3.in‚0) disk./.usagePercent‚77.84269562785008 disk.dm-0.in‚ 14357.328125Àoç›ñ‚(: çìçÔÁ½ëx gpu.0.gpu‚5.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.674799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª¹LЂò‚(:÷ çìçÔÜ—¤y disk.sda.out‚ 159.5703125! disk.nvme0n1.in‚ 14494.1796875 cpu‚25.061557125777174% disk./.usageGB‚1159.9123992919922 disk.dm-0.in‚ 14494.1796875 disk.dm-1.in‚0 disk.dm-1.out‚ 159.5703125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.73046875 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 7.527307e+06) disk./.usagePercent‚77.84269434781241# disk.nvme0c0n1.in‚ 14494.1796875 disk.sda3.out‚ 159.5703125' disk.nvme0n1.out‚122184.13427734375 disk.sda2.out‚0# proc.memory.rssMB‚ 3104.75390625) proc.memory.percent‚18.94991397857666) disk.nvme0c0n1.out‚122184.13427734375 disk.drbd1260.in‚0 disk.sda1.in‚0# memory_percent‚89.3726110458374) proc.memory.availableMB‚ 1741.19140625 proc.cpu.threads‚17 network.recv‚ 1.668438e+06$ disk.dm-0.out‚122184.13427734375 disk.sda.in‚0C&Ž‘ó‚(:† öìçÔ­Ç›y gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚143.814' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…û—‚ô‚(:÷ öìçÔð˜´z) disk./.usagePercent‚77.84111938946177& disk.nvme0n1.out‚122482.2841796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 3109.87890625* proc.memory.percent‚18.981194496154785 network.recv‚ 1.671781e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.73046875 disk.sda1.out‚0 disk.dm-0.in‚ 14494.1796875 disk.dm-1.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1736.19140625 network.sent‚ 7.532259e+06$ disk./.usageGB‚1159.888931274414 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 14494.1796875# disk.dm-0.out‚122482.2841796875 disk.sda.out‚ 159.93359375 disk.sda3.out‚ 159.93359375# memory_percent‚89.4031286239624! disk.nvme0n1.in‚ 14494.1796875 disk.dm-1.out‚ 159.93359375 cpu‚25.062173515760257 proc.cpu.threads‚17( disk.nvme0c0n1.out‚122482.2841796875s”¶©|õ‚(:ñ …íçÔæ…y$ disk./.usageGB‚1159.888931274414 disk.dm-1.in‚0& disk.nvme0n1.out‚122901.1376953125 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 14494.1796875( disk.nvme0c0n1.out‚122901.1376953125 disk.dm-0.in‚ 14494.1796875 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚89.43517208099365# proc.memory.rssMB‚ 3115.00390625 cpu‚25.06285730654085) disk./.usagePercent‚77.84111938946177 disk.sda3.in‚0 disk.sda3.out‚ 160.3359375# disk.dm-0.out‚122901.1376953125) proc.memory.availableMB‚ 1730.94140625) proc.memory.percent‚19.01247501373291 proc.cpu.threads‚17 network.sent‚ 7.536614e+06 disk.dm-1.out‚ 160.3359375 disk.sda.in‚0 disk.sda.out‚ 160.3359375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.734375! disk.nvme0n1.in‚ 14494.1796875 disk.sda2.out‚0 network.recv‚ 1.676579e+06Ñy«‘ö‚(:† …íçÔŽ¾áy gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Æ÷‚(:… ”íçÔ•‘“y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£7C°}ø‚(:ò ”íçÔ쩲y disk.sda1.in‚0 disk.sda1.out‚0$ disk.dm-0.out‚123262.24658203125 cpu‚25.06338130434284 network.recv‚ 1.679766e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚0.734375! disk.nvme0n1.in‚ 14494.1796875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 14494.1796875 disk.sda.out‚ 160.9296875# memory_percent‚89.4618034362793) disk./.usagePercent‚77.84112066949943 disk.dm-0.in‚ 14494.1796875 disk.dm-1.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 1726.578125# proc.memory.rssMB‚ 3119.50390625% disk./.usageGB‚1159.8889503479004 disk.sda3.in‚0 disk.sda3.out‚ 160.9296875 disk.sda2.out‚0) disk.nvme0c0n1.out‚123262.24658203125 disk.dm-1.out‚ 160.9296875) proc.memory.percent‚19.03994083404541 proc.cpu.threads‚17 network.sent‚ 7.541348e+06' disk.nvme0n1.out‚123262.24658203125~-}2ù- _wandbruntime‚5240  _runtime‚5240î°ë¾Åú¿ % _timestamp‚1.7884750337400405e+09 _runtime‚5240.496596863 _step‚238 % lr-AdamW/pg1‚0.000611580733919396 % lr-AdamW/pg2‚0.000611580733919396 trainer/global_step‚2949i@ÊÁÊûÄ % lr-AdamW/pg2‚0.000611580733919396 trainer/global_step‚2949 % _timestamp‚1.7884750337400405e+09 _runtime‚5240.496596863 _step‚238 % lr-AdamW/pg1‚0.000611580733919396îâ´£¢2ü- _wandbruntime‚5240  _runtime‚5240cÝA(ýúO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467253634438144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚463293029636727 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚463289395546079 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62222055 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7178968521216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938560 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467253660829696 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66193566 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73763860 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135985915 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136012260 _runtime‚5240.500376156 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚456123033861872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467253634438144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7182579181568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456123008433904 % _timestamp‚1.7884750337433164e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73763860 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚474436240011264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7166387112175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73790199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474436240011264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39331477 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚432838369195520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚422950150641664 trainer/global_step‚2949 _step‚239 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7178968521216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467253660829696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73790199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474432602959360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62222055 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9888341695488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62222061 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432838492337152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7169995774855 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39331472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9888269756928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7182579181568 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938560 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467253660829696 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66193566 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73763860 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135985915 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136012260 _runtime‚5240.500376156 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚456123033861872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467253634438144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7182579181568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456123008433904 % _timestamp‚1.7884750337433164e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73763860 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚474436240011264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7166387112175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73790199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474436240011264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39331477 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚432838369195520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚422950150641664 trainer/global_step‚2949 _step‚239 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7178968521216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467253660829696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allb4SSëocated‚73790199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474432602959360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62222055 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9888341695488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62222061 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚432838492337152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7169995774855 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39331472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9888269756928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7182579181568 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7181379002880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26871481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73814792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚467407438388736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474588791000064 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128320 % _timestamp‚1.7884750361057622e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136057277 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7184989676544 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136057277 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73788427 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7168793415461 trainer/global_step‚2949 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474592428065280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136030906 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925248 _step‚240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9891665865728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66215086 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚432980983768064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891593940480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62242485 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26870809 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423089441030656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚456273337675040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 5242.86706225 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚467407411997184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532MÓ'ˆN F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚474592428065280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136030906 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925248 _step‚240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9891665865728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66215086 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚432980983768064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9891593940480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62242485 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26870809 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423089441030656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚456273337675040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 5242.86706225 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚467407411997184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7181379002880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚463445765181357 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66215763 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474592428065280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚432981106896384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7181379002880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26871481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73814792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚467407438388736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474588791000064 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128320 % _timestamp‚1.7884750361057622e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136057277 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7184989676544 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136057277 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73788427 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7168793415461 trainer/global_step‚2949ð™ÕÚò2‚- _wandbruntime‚5242  _runtime‚5242?3_TƒÎ $ _timestamp‚1.788475036114424e+09 /loss/FRt3/assignment_loss‚5.556114276250203 2loss/SRqqt1/assignment_loss‚1.6217619981784541 1loss/SRqqt4/assignment_loss‚2.097981973336293 0loss/FBt2/assignment_loss‚0.2826992257389464 1loss/SRqqt3/detection_loss‚2.6156564396003197 trainer/global_step‚2949 2loss/SRqqt3/assignment_loss‚1.7702362187977494 /loss/FRt1/detection_loss‚1.6256614072453999 .loss/FRt2/detection_loss‚1.971716502538094 -loss/FBt3/detection_loss‚2.09392419308424 /loss/FBt4/detection_loss‚2.4821548591489377 _step‚241 0loss/FRt1/assignment_loss‚3.0266402304172515 .loss/FRt3/detection_loss‚4.157086839278539 /loss/FRt4/detection_loss‚3.2008062645241067 1loss/SRqqt1/detection_loss‚1.0531801476000666 .loss/FBt1/detection_loss‚5.873760938644409 /loss/FRt4/assignment_loss‚4.342394872947976 0loss/FBt3/assignment_loss‚0.6159006270521786 /loss/FRt2/assignment_loss‚2.778427840425418 1loss/FBt1/assignment_loss‚0.04811527952551842 1loss/SRqqt2/detection_loss‚2.9892222695052624 _runtime‚ 5242.87103595 1loss/FBt4/assignment_loss‚0.40000023111307476 1loss/SRqqt2/assignment_loss‚2.308049738407135 1loss/SRqqt4/detection_loss‚2.8957629467432318 %loss/total_loss‚56.63211212864307 .loss/FBt2/detection_loss‚0.824856808540267 epoch‚2sæt0Y„Ó /loss/FRt2/assignment_loss‚2.778427840425418 1loss/FBt1/assignment_loss‚0.04811527952551842 1loss/SRqqt2/detection_loss‚2.9892222695052624 _runtime‚ 5242.87103595 1loss/FBt4/assignment_loss‚0.40000023111307476 1loss/SRqqt2/assignment_loss‚2.308049738407135 1loss/SRqqt4/detection_loss‚2.8957629467432318 %loss/total_loss‚56.63211212864307 .loss/FBt2/detection_loss‚0.824856808540267 epoch‚2 $ _timestamp‚1.788475036114424e+09 /loss/FRt3/assignment_loss‚5.556114276250203 2loss/SRqqt1/assignment_loss‚1.6217619981784541 1loss/SRqqt4/assignment_loss‚2.097981973336293 0loss/FBt2/assignment_loss‚0.2826992257389464 1loss/SRqqt3/detection_loss‚2.6156564396003197 trainer/global_step‚2949 2loss/SRqqt3/assignment_loss‚1.7702362187977494 /loss/FRt1/detection_loss‚1.6256614072453999 .loss/FRt2/detection_loss‚1.971716502538094 -loss/FBt3/detection_loss‚2.09392419308424 /loss/FBt4/detection_loss‚2.4821548591489377 _step‚241 0loss/FRt1/assignment_loss‚3.0266402304172515 .loss/FRt3/detection_loss‚4.157086839278539 /loss/FRt4/detection_loss‚3.2008062645241067 1loss/SRqqt1/detection_loss‚1.0531801476000666 .loss/FBt1/detection_loss‚5.873760938644409 /loss/FRt4/assignment_loss‚4.342394872947976 0loss/FBt3/assignment_loss‚0.6159006270521786ñÇŽV…‚(:’ £íçÔ°°îy gpu.0.gpu‚76.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚180.145' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.028999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑȠ‚(:õ £íçÔÆàýz network.recv‚ 1.684484e+06 disk.sda2.out‚0$ disk.nvme0c0n1.in‚14615.38671875 disk.dm-0.in‚14615.38671875 disk.dm-1.in‚0 network.sent‚ 7.560779e+06) disk./.usagePercent‚77.84270305206857% disk./.usageGB‚1159.9125289916992 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚123607.85498046875 disk.sda3.out‚ 161.53125$ disk.dm-0.out‚123607.85498046875 disk.sda.in‚0 disk.sda.out‚ 161.53125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.73828125# proc.memory.rssMB‚ 3124.62890625 cpu‚25.063935710456043" disk.nvme0n1.in‚14615.38671875' disk.nvme0n1.out‚123607.85498046875 disk.dm-1.out‚ 161.53125$ memory_percent‚89.49284553527832( proc.memory.availableMB‚ 1721.4921875* proc.memory.percent‚19.071221351623535 proc.cpu.threads‚17]7À-‘‡‚(:† ²íçÔÖÙÐx gpu.0.gpu‚34.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚136.146' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.2292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â]†ˆ‚(:û ²íçÔ«¸€y proc.cpu.threads‚17 network.recv‚ 1.690252e+06 disk.dm-1.in‚0 disk.sda.out‚ 162.09765625 disk.sda3.out‚ 162.09765625 disk.drbd1260.out‚ 0.73828125' disk.nvme0n1.out‚123763.40185546875 cpu‚25.06456916111312( disk./.usagePercent‚77.8427068921816 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚14615.38671875 disk.dm-1.out‚ 162.09765625" disk.nvme0n1.in‚14615.38671875$ memory_percent‚89.52031135559082 network.sent‚ 7.655393e+06% disk./.usageGB‚1159.9125862121582 disk.sda1.out‚0$ disk.nvme0c0n1.in‚14615.38671875) disk.nvme0c0n1.out‚123763.40185546875$ disk.dm-0.out‚123763.40185546875 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1716.9921875" proc.memory.rssMB‚ 3129.1171875* proc.memory.percent‚19.098615646362305Øú®ˆš‰‚(: ÁíçÔïÓ¨x gpu.0.gpu‚7.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚259.708' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.94160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¾ñHpŠ‚(:å ÁíçÔ¥£ëz disk.dm-1.out‚162.5 disk.sda.out‚162.5% disk./.usageGB‚1159.9123725891113 disk.sda3.out‚162.5 disk.drbd1260.out‚ 0.7421875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚124037.35009765625 disk.dm-0.in‚14615.38671875" disk.nvme0n1.in‚14615.38671875 disk.sda.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚14615.38671875$ memory_percent‚89.54312801361084) proc.memory.availableMB‚ 1713.25390625( proc.memory.percent‚19.1220760345459 disk.sda2.in‚0$ disk.dm-0.out‚124037.35009765625" proc.memory.rssMB‚ 3132.9609375 cpu‚25.06530123001431 proc.cpu.threads‚17 disk.drbd1260.in‚0' disk.nvme0n1.out‚124037.35009765625 network.sent‚ 7.661529e+06 network.recv‚ 1.694715e+06) disk./.usagePercent‚77.84269255575967 disk.dm-1.in‚0ÌaË|’‹‚(:‡ ÐíçÔ”‡Ûx gpu.0.gpu‚52.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚200.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.1708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k“Àº†Œ‚(:û ÐíçÔ¢Çîx disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7421875 network.sent‚ 7.666581e+06 network.recv‚ 1.69814e+06% disk./.usageGB‚1159.8888664245605 disk.sda3.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 162.88671875 disk.sda3.out‚ 162.88671875' disk.nvme0n1.out‚124456.11181640625 disk.sda2.out‚0) disk.nvme0c0n1.out‚124456.11181640625( proc.memory.availableMB‚ 1707.9140625" proc.memory.rssMB‚ 3138.0859375 cpu‚25.065880796491673) disk./.usagePercent‚77.84111503733368$ disk.dm-0.out‚124456.11181640625 disk.dm-1.in‚0 disk.dm-1.out‚ 162.88671875" disk.nvme0n1.in‚14615.38671875$ disk.nvme0c0n1.in‚14615.38671875$ memory_percent‚89.57571983337402 disk.dm-0.in‚14615.38671875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda1.out‚0* proc.memory.percent‚19.153356552124023ZM8Ž'‚(:ø ßíçÔ“Æ‘x disk.sda1.out‚0( disk.nvme0c0n1.out‚124875.1083984375* proc.memory.percent‚19.180727005004883 cpu‚25.066398558166473 proc.cpu.threads‚17 disk.sda3.out‚ 163.48046875 disk.drbd1260.in‚0 network.sent‚ 7.672795e+06$ disk./.usageGB‚1159.888828277588# disk.dm-0.out‚124875.1083984375 disk.dm-1.out‚ 163.48046875 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 163.48046875" disk.nvme0n1.in‚14633.14453125 disk.sda2.out‚0 disk.sda1.in‚0$ di½U°~\sk.nvme0c0n1.in‚14633.14453125# memory_percent‚89.6024227142334( proc.memory.availableMB‚ 1703.5390625 disk.drbd1260.out‚ 0.74609375 network.recv‚ 1.703144e+06 disk.dm-0.in‚14633.14453125" proc.memory.rssMB‚ 3142.5703125) disk./.usagePercent‚77.84111247725833& disk.nvme0n1.out‚124875.1083984375 disk.sda2.in‚0»4/d‘Ž‚(:† ßíçÔÙ†Íx gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚197.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.4684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞU§|›‚(: îíçÔϳóx gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ÛÐt‚(:é îíçԌʩy network.sent‚ 7.677745e+06) disk./.usagePercent‚77.84268743560898$ disk.nvme0c0n1.in‚14732.91796875# disk.dm-0.out‚125220.7060546875& disk.nvme0n1.out‚125220.7060546875 disk.sda2.out‚0 proc.memory.rssMB‚ 3144.8125$ disk./.usageGB‚1159.912296295166( disk.nvme0c0n1.out‚125220.7060546875 disk.dm-0.in‚14732.91796875 disk.sda.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚14732.91796875 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda.out‚ 163.84375 disk.sda3.in‚0 disk.sda3.out‚ 163.84375 disk.drbd1260.out‚ 0.74609375 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚89.61563110351562 network.recv‚ 1.706664e+06 disk.dm-1.out‚ 163.84375$ proc.memory.availableMB‚1701.375* proc.memory.percent‚19.194412231445312 cpu‚25.066941051780457nÊ=r‘‚(:ç ýíçÔË­‰z$ memory_percent‚89.61665630340576) disk./.usagePercent‚77.84268333948845% disk./.usageGB‚1159.9122352600098 disk.sda3.out‚ 164.3515625 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚14732.91796875& disk.nvme0c0n1.out‚125485.62890625* proc.memory.percent‚19.194412231445312 cpu‚25.06693325748838 network.sent‚ 7.683973e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 164.3515625 disk.sda.in‚0 disk.sda.out‚ 164.3515625 disk.drbd1260.out‚0.75$ disk.nvme0n1.out‚125485.62890625 disk.sda2.in‚0 network.recv‚ 1.71143e+06! disk.dm-0.out‚125485.62890625) proc.memory.availableMB‚ 1701.20703125 proc.memory.rssMB‚ 3144.8125 proc.cpu.threads‚17 disk.dm-0.in‚14732.91796875 disk.sda3.in‚0" disk.nvme0n1.in‚14732.91796875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0æA·Wœ’‚(:‘ ýíçÔÉí¢| gpu.0.gpu‚34.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚182.126' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.425200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p4wÅ2“- _wandbruntime‚5341  _runtime‚5341OÏŽïÇ”Á & lr-AdamW/pg1‚0.0006092854846147969 & lr-AdamW/pg2‚0.0006092854846147969 trainer/global_step‚2999 % _timestamp‚1.7884751345552955e+09 _runtime‚5341.312962548 _step‚242ñ–qÌ•Æ & lr-AdamW/pg2‚0.0006092854846147969 trainer/global_step‚2999 % _timestamp‚1.7884751345552955e+09 _runtime‚5341.312962548 _step‚242 & lr-AdamW/pg1‚0.0006092854846147969òì­Ü2–- _wandbruntime‚5341  _runtime‚5341ØOj­(—üO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482242019408384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚727 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67274827 _step‚243 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7290324530363 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138238817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚482242019408384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439969201064448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63243261 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74995562 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63243255 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482245656460288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2999 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74995562 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10054530973184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 % _timestamp‚1.7884751345598314e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7299507018240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10054456937472 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 _runtime‚5341.317523986 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39971722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7286715867683 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7303117678592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67275554 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474942538781696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75021901 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474942512390144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚429914670091264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚429914618888192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463639495918672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39971727 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27303827 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75021901 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7303117678592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27303105 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474942512390144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482242019408384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚727 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67274827 _step‚243 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7290324530363 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138238817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚482242019408384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439969201064448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63243261 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74995562 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63243255 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482245656460288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2999 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74995562 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10054530973184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 % _timestamp‚1.7884751345598314e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7299507018240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10054456937472 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 _runtime‚5341.317523986 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39971722 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚722 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7286715867683 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7303117678592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67275554 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474942538781696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75021901 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474942512390144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚429914670091264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚429914618888192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463639495918672 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39971727 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27303827 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75021901 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7303117678592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27303105 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚474942512390144 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63263679 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138283808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63263685 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10057766561280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138283808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27311799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7301917499904 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67296326 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39984527 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75020129 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚482401844514304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚475096316340736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7305528173568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138310179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440111801063936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463789799731840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7292730833857 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430053909277184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463789825159808 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10057840583680 trainer/global_step‚2999 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75046494 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocate.žª ¨d‚3719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39984532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚471082555993665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 _step‚244 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63263685 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7305528173568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚471078921902809 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚475096289949184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440111675838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75046494 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚475096316340736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365/ü'›‰N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10057766561280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138283808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27311799 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7301917499904 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67296326 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39984527 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75020129 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚482401844514304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚475096316340736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7305528173568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138310179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440111801063936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463789799731840 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7292730833857 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430053909277184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463789825159808 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10057840583680 trainer/global_step‚2999 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75046494 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39984532 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚471082555993665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 _step‚244 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63263685 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7305528173568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚471078921902809 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚475096289949184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440111675838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75046494 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚475096316340736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 _runtime‚ 5343.60593379 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75020129 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.788475136846476e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63263679 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138283808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63263685 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472ô5:Z2œ- _wandbruntime‚5343  _runtime‚5343ª3øŠWÑ 2loss/SRqqt3/assignment_loss‚2.1112562139567577 .loss/FRt1/detection_loss‚1.656799332108295 0loss/SRqqt1/detection_loss‚1.144381397802915 /loss/FBt4/detection_loss‚2.4232444246610005 1loss/FBt2/assignment_loss‚0.25000320774551543 /loss/FRt2/detection_loss‚2.0870854331610293 2loss/SRqqt1/assignment_loss‚1.3878166692863618 .loss/FRt3/detection_loss‚3.372247717597268 1loss/SRqqt4/detection_loss‚3.1457136422395706 .loss/FBt1/detection_loss‚4.903936266899109 .loss/FBt3/detection_loss‚2.453575611114502 %loss/total_loss‚53.45099522368043 % _timestamp‚1.7884751368529499e+09 trainer/global_step‚2999 2loss/SRqqt2/assignment_loss‚2.3268672092394396 1loss/SRqqt4/assignment_loss‚2.855850499868393 1loss/FBt1/assignment_loss‚0.22680805623531342 0loss/FBt3/assignment_loss‚0.2719411249617176 .loss/FRt4/detection_loss‚2.781742373579427 /loss/FRt3/assignment_loss‚5.437090321020647 0loss/SRqqt2/detection_loss‚2.403294086456299 1loss/SRqqt3/detection_loss‚2.2687623845903495 /loss/FBt2/detection_loss‚0.8546688026965481 epoch‚2 _runtime‚5343.609563592 /loss/FRt2/assignment_loss‚2.910434670173205 0loss/FRt4/assignment_loss‚3.6830056780262996 1loss/FBt4/assignment_loss‚0.31461836931606135 _step‚245 /loss/FRt1/assignment_loss‚2.179851730944405.¬L\žÖ /loss/FRt1/assignment_loss‚2.179851730944405 2loss/SRqqt3/assignment_loss‚2.1112562139567577 .loss/FRt1/detection_loss‚1.656799332108295 0loss/SRqqt1/detection_loss‚1.144381397802915 /loss/FBt4/detection_loss‚2.4232444246610005 1loss/FBt2/assignment_loss‚0.25000320774551543 /loss/FRt2/detection_loss‚2.0870854331610293 2loss/SRqqt1/assignment_loss‚1.3878166692863618 .loss/FRt3/detection_loss‚3.372247717597268 1loss/SRqqt4/detection_loss‚3.1457136422395706 .loss/FBt1/detection_loss‚4.903936266899109 .loss/FBt3/detection_loss‚2.453575611114502 %loss/total_loss‚53.45099522368043 % _timestamp‚1.7884751368529499e+09 trainer/global_step‚2999 2loss/SRqqt2/assignment_loss‚2.3268672092394396 1loss/SRqqt4/assignment_loss‚2.855850499868393 1loss/FBt1/assignment_loss‚0.22680805623531342 0loss/FBt3/assignment_loss‚0.2719411249617176 .loss/FRt4/detection_loss‚2.781742373579427 /loss/FRt3/assignment_loss‚5.437090321020647 0loss/SRqqt2/detection_loss‚2.403294086456299 1loss/SRqqt3/detection_loss‚2.2687623845903495 /loss/FBt2/detection_loss‚0.8546688026965481 epoch‚2 _runtime‚5343.609563592 /loss/FRt2/assignment_loss‚2.910434670173205 0loss/FRt4/assignment_loss‚3.6830056780262996 1loss/FBt4/assignment_loss‚0.31461836931606135 _step‚245õñ„®sŸ‚(:è ŒîçÔû¸¥{ disk.drbd1260.out‚0.75# disk.nvme0n1.out‚125737.3046875 disk.sda2.out‚0 disk.sda1.out‚0 network.sent‚ 7.688808e+06$ disk.nvme0c0n1.in‚14732.91796875 disk.dm-1.in‚0" disk.nvme0n1.in‚14732.91796875 disk.sda1.in‚0 proc.memory.rssMB‚ 3144.8125 cpu‚25.066966817827325 network.recv‚ 1.714523e+06$ disk./.usageGB‚1159.889030456543 disk.dm-0.in‚14732.91796875 disk.dm-0.out‚125737.3046875 disk.dm-1.out‚ 164.74609375 disk.sda3.in‚0 proc.cpu.threads‚17% disk.nvme0c0n1.out‚125737.3046875 disk.sda.out‚ 164.74609375 disk.sda2.in‚0$ memory_percent‚89.61679935455322) proc.memory.availableMB‚ 1701.18359375* proc.memory.percent‚19.194412231445312 disk.sda3.out‚ 164.74609375) disk./.usagePercent‚77.84112604565765 disk.sda.in‚0 disk.drbd1260.in‚0ü>(¯’ ‚(:‡ ŒîçÔúØÖ{ gpu.0.gpu‚52.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚209.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.9976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü“]œ¡‚(:‘ ›îçÔŠÑÍ{ gpu.0.gpu‚17.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.775' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.955000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙEoƒ¢‚(:ø ›îçÔîåŸ| disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 165.3203125 disk.sda.in‚0 disk.sda.out‚ 165.3203125$ memory_percent‚89.61679935455322 network.sent‚ 7.797881e+06 disk.sda3.out‚ 165.3203125 disk.drbd1260.out‚ 0.75390625" disk.nvme0n1.in‚14732.91796875' disk.nvme0n1.out‚126155.10205078125) proc.memory.availableMB‚ 1701.18359375 proc.memory.rssMB‚ 3144.8125 cpu‚25.067048206588506 network.recv‚ 1.721264e+06$ disk.dm-0.out‚126155.10205078125* proc.memory.percent‚19.194412231445312 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda1.in‚0) disk./.usagePercent‚77.84112476561998 disk.sda3.in‚0 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚14732.91796875) disk.nvme0c0n1.out‚126155.10205078125 disk.dm-0.in‚14732.91796875% disk./.usageGB‚1159.8890113830566p˜9|›£‚(: ªîçÔÖˆ¦w gpu.0.gpu‚88.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚221.2' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.239999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªí2dw¤‚(:ì ªîçÔ†ð”x" disk.nvme0n1.in‚14855.05078125% disk./.usageGB‚1159.8655471801758 disk.sda1.in‚0$ disk.nvme0c0n1.in‚14855.05078125 disk.dm-1.in‚0 disk.sda3.out‚165.875 disk.drbd1260.in‚0' disk.nvme0n1.out‚126597.87548828125 disk.sda2.in‚0) disk./.usagePercent‚77.83955006327687 disk.dm-0.in‚14855.05078125 disk.sda.out‚165.875$ memory_percent‚89.61679935455322) proc.memory.availableMB‚ 1701.18359375 proc.memory.rssMB‚ 3144.8125 disk.dm-1.out‚165.875 disk.drbd1260.out‚ 0.75390625 disk.sda2.out‚0* proc.memory.percent‚19.194412231445312 cpu‚25.067085897569836 proc.cpu.threads‚17 network.sent‚ 7.804596e+06 network.recv‚ 1.724544e+06 disk.sda1.out‚0) disk.nvme0c0n1.out‚126597.87548828125$ disk.dm-0.out‚126597.87548828125 disk.sda.in‚0 disk.sda3.in‚00iG—}¥‚(:ò ¹îçÔÏœÇy network.sent‚ 7.809119e+06 network.recv‚ 1.729617e+06$ disk.nvme0c0n1.in‚14855.05078125' disk.nvme0c0n1.out‚127004.677734375 disk.dm-0.in‚14855.05078125" disk.dm-0.out‚127004.677734375$ memory_percent‚89.61682319641113( proc.memory.availableMB‚ 1701.1796875) disk./.usagePercent‚77.84270151602337 disk.sda3.out‚ 166.19140625 disk.drbd1260.in‚0 proc.memory.rssMB‚ 3144.8125 cpu‚25.06711720709984 disk.sda.out‚ 166.19140625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.7578125 disk.sda2.in‚0 disk.sda1.in‚0* proc.memory.percent‚19.194412231445312 proc.cpu.threads‚17% disk.nvme0n1.out‚127004.677734375% disk./.usageGB‚1159.9125061035156 disk.dm-1.in‚0 disk.dm-1.out‚ 166.19140625 disk.sda.in‚0" disk.nvme0n1.in‚14855.05078125 disk.sda2.out‚0 disk.sda1.out‚0áóБ¦‚(:† ¹îçÔ©çÌy gpu.0.gpu‚14.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚34.0 gpu.0.powerWatts‚101.483' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2966 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uq… n§‚(:ã ÈîçÔ§¦x disk.sda.out‚166.5625" disk.nvme0n1.in‚14855.05078125 disk.sda2.out‚0 proc.memory.rssMB‚ 3144.8125) disk./.usagePercent‚77.84270151602337% disk./.usageGB‚1159.9125061035156" disk.dm-0.out‚127242.060546875 disk.sda3.in‚0% disk.nvme0n1.out‚127242.060546875 disk.sda2.in‚0 disk.sda1.out‚0 cpu‚25.067083124753225 network.sent‚ 7.813852e+06 network.recv‚ 1.73273e+06' disk.nvme0c0n1.out‚127242.060546875 disk.dm-1.in‚0 disk.sda3.out‚166.5625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7578125$ memory_percent‚89.61677551269531$ disk.nvme0c0n1.in‚14855.05078125 disk.dm-1.out‚166.5625 disk.sda.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 1701.1875* proc.memory.percent‚19.194412231445312 proc.cpu.threads‚17 disk.dm-0.in‚14855.05078125ð¡Ã‘¨‚(:† ÈîçÔ¼öÚy gpu.0.gpu‚29.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚212.113' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.4226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(„ƒ©‚(:ø ×îçÔ€Ùy) disk.nvme0c0n1.out‚127434.82080078125 disk.dm-0.in‚14855.05078125 disk.sda.in‚0 disk.sda.out‚ 167.12890625 disk.sda3.in‚0 disk.drbd1260.out‚ 0.76171875" disk.nvme0n1.in‚14855.05078125 network.recv‚ 1.7378e+06% disk./.usageGB‚1159.9124870300293$ disk.nvme0c0n1.in‚14855.05078125$ disk.dm-0.out‚127434.82080078125 disk.sda3.out‚ 167.12890625' disk.nvme0n1.out‚127434.82080078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 167.12890625 disk.drbd1260.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1701.18359375 proc.memory.rssMB‚ 3144.8125* proc.memory.percent‚19.194412231445312 proc.cpu.threads‚17) disk./.usagePercent‚77.84270023598569 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚89.61679935455322 cpu‚25.06716393066371 network.sent‚ 7.820287e+064bˆnœª‚(:‘ ×îçÔÑÔy gpu.0.gpu‚43.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.016' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.603199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õªàù’«‚(:‡ æîçÔ²Ÿ¡y gpu.0.gpu‚55.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚205.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.0754 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò!¾—}¬‚(:ò æîçÔ›Ù©z! disk.nvme0c0n1.in‚ 14975.40625* proc.memory.percent‚19.194459915161133 cpu‚25.067106461974593 proc.cpu.threads‚17) disk./.usagePercent‚77.84268589956378 disk.dm-1.out‚ 167.5390625 disk.sda.in‚0 disk.sda.out‚ 167.5390625 disk.sda3.in‚0 disk.sda3.out‚ 167.5390625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 14975.40625 network.sent‚ 7.825113e+06 network.recv‚ 1.741191e+06' disk.nvme0n1.out‚127613.29736328125 disk.sda2.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚127613.29736328125) proc.memory.availableMB‚ 1700.93359375" proc.memory.rssMB‚ 3144.8203125$ disk.dm-0.out‚127613.29736328125 disk.drbd1260.out‚ 0.76171875 disk.sda1.out‚0$ memory_percent‚89.61832523345947% disk./.usageGB‚1159.9122734069824 disk.dm-0.in‚ 14975.40625 disk.dm-1.in‚0 disk.sda2.out‚0ÅÒª2­- _wandbruntime‚5445  _runtime‚5445{†ÃÇ®Á & lr-AdamW/pg2‚0.0006069311836286581 trainer/global_step‚3049 % _timestamp‚1.7884752388633423e+09 _runtime‚5445.620419042 _step‚246 & lr-AdamW/pg1‚0.0006069311836286581×éŠÌ¯Æ & lr-AdamW/pg1‚0.0006069311836286581 & lr-AdamW/pg2‚0.0006069311836286581 trainer/global_step‚3049 % _timestamp‚1.7884752388633423e+09 _runtime‚5445.620419042 _step‚246ö:úöL2°- _wandbruntime‚5445  _runtime‚54455¹^½T!±üO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64264461 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27746267 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 trainer/global_step‚3049 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚717 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40611977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚447099876566528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚247 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚490051435857408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚490055072909312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚436879189540864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚722 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7423656175616 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚482631416733696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68358961 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719ï¡ï(²P UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478562977170695 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471155957975472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚447099751327744 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10220687025664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76253603 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7420045515264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚436879138337792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚490051435857408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚490055072909312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚436879189540864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚722 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7423656175616 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚482631416733696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68358961 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64264461 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27746267 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 trainer/global_step‚3049 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚717 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40611977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚447099876566528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚247 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68379936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚482785167901184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478715712715117 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64284885 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚5448.104383496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚490211260963328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140563081 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68380631 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10223952892416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76278196 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437018479929856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64284879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40624777 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478719346805973 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788475241344502e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437018428726784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76278196 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7422455996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7413059589365 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 trainer/global_step‚3049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0`Î\'µŠN WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478715712715117 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64284885 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚5448.104383496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚490211260963328 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140563081 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68380631 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10223952892416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76278196 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437018479929856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64284879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40624777 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478719346805973 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788475241344502e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437018428726784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76278196 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7422455996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7413059589365 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 trainer/global_step‚3049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7426066670592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚482785194292736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚482785167901184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27755159 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76251831 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚490211260963328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 _step‚248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447242381619200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68379936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚482785167901184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0øÂn$2¶- _wandbruntime‚5448  _runtime‚5448÷Ý‘›·Õ _runtime‚5448.108216732 1loss/FBt2/assignment_loss‚0.24385035919448178 .loss/FRt3/detection_loss‚3.612330950796604 0loss/SRqqt3/detection_loss‚2.278781563946695 /loss/FBt3/detection_loss‚2.4941351181930966 trainer/global_step‚3049 0loss/SRqqt2/detection_loss‚2.241651728749275 1loss/FBt3/assignment_loss‚0.49843370579441804 /loss/FRt1/detection_loss‚1.6691749893871146 0loss/SRqqt4/detection_loss‚2.864487910270691 2loss/SRqqt2/assignment_loss‚1.7887106463313103 % _timestamp‚1.7884752413518734e+09 0loss/FBt1/assignment_loss‚0.0779720911680973 1loss/SRqqt1/detection_loss‚1.0883589352504546 /loss/FBt.­bÀ1/detection_loss‚3.5141662508249283 /loss/FBt4/detection_loss‚2.2736020565032957 /loss/FRt3/assignment_loss‚4.862160876393318 /loss/FBt2/detection_loss‚1.1089020153241498 /loss/FRt2/assignment_loss‚3.452057288502747 1loss/SRqqt1/assignment_loss‚1.486048306988067 2loss/SRqqt3/assignment_loss‚2.3036794048367124 2loss/SRqqt4/assignment_loss‚2.7065126955509187 /loss/FRt2/detection_loss‚2.1865750541383364 /loss/FRt4/detection_loss‚2.7537639704910486 _step‚249 0loss/FRt1/assignment_loss‚2.6663780882954597 0loss/FRt4/assignment_loss‚3.7098596418226086 1loss/FBt4/assignment_loss‚0.44099839888513087 %loss/total_loss‚52.32259204763896 epoch‚27”7ò`¸Ú /loss/FRt2/assignment_loss‚3.452057288502747 1loss/SRqqt1/assignment_loss‚1.486048306988067 2loss/SRqqt3/assignment_loss‚2.3036794048367124 2loss/SRqqt4/assignment_loss‚2.7065126955509187 /loss/FRt2/detection_loss‚2.1865750541383364 /loss/FRt4/detection_loss‚2.7537639704910486 _step‚249 0loss/FRt1/assignment_loss‚2.6663780882954597 0loss/FRt4/assignment_loss‚3.7098596418226086 1loss/FBt4/assignment_loss‚0.44099839888513087 %loss/total_loss‚52.32259204763896 epoch‚2 _runtime‚5448.108216732 1loss/FBt2/assignment_loss‚0.24385035919448178 .loss/FRt3/detection_loss‚3.612330950796604 0loss/SRqqt3/detection_loss‚2.278781563946695 /loss/FBt3/detection_loss‚2.4941351181930966 trainer/global_step‚3049 0loss/SRqqt2/detection_loss‚2.241651728749275 1loss/FBt3/assignment_loss‚0.49843370579441804 /loss/FRt1/detection_loss‚1.6691749893871146 0loss/SRqqt4/detection_loss‚2.864487910270691 2loss/SRqqt2/assignment_loss‚1.7887106463313103 % _timestamp‚1.7884752413518734e+09 0loss/FBt1/assignment_loss‚0.0779720911680973 1loss/SRqqt1/detection_loss‚1.0883589352504546 /loss/FBt1/detection_loss‚3.5141662508249283 /loss/FBt4/detection_loss‚2.2736020565032957 /loss/FRt3/assignment_loss‚4.862160876393318 /loss/FBt2/detection_loss‚1.1089020153241498ùòx2^‘¹‚(:† õîçÔÕüw gpu.0.gpu‚93.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.545 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Ó‡©zº‚(:ï õîçÔ¼øˆz network.sent‚ 7.830377e+06 network.recv‚ 1.745065e+06 disk.sda1.out‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1701.11328125" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133) disk./.usagePercent‚77.84111708539395' disk.nvme0n1.out‚128056.36962890625$ disk.dm-0.out‚128056.36962890625 disk.dm-1.in‚0 disk.dm-1.out‚ 167.7578125 disk.sda.in‚0 disk.sda.out‚ 167.7578125 disk.sda3.out‚ 167.7578125% disk./.usageGB‚1159.8888969421387 disk.nvme0n1.in‚ 14975.40625 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 14975.40625) disk.nvme0c0n1.out‚128056.36962890625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.765625# memory_percent‚89.6172285079956 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 14975.40625 cpu‚25.067107279385525 proc.cpu.threads‚177â6»‚(:’ „ïçÔ·…ˆw gpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚254.617' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.923399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÍ2$|¼‚(:ñ „ïçÔ¥êúw' proc.memory.availableMB‚ 1701.109375" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133 disk.sda1.in‚0 network.recv‚ 1.750145e+06 disk.sda3.out‚ 168.11328125 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 14975.40625 disk.dm-0.in‚ 14975.40625 disk.dm-1.in‚0 disk.dm-1.out‚ 168.11328125 network.sent‚ 7.937755e+06 disk.sda3.in‚0 cpu‚25.067215652365125 proc.cpu.threads‚17 disk.sda1.out‚0% disk./.usageGB‚1159.2610626220703 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 14975.40625' disk.nvme0n1.out‚128572.56103515625 disk.sda2.in‚0$ disk.dm-0.out‚128572.56103515625$ memory_percent‚89.61725234985352) disk./.usagePercent‚77.79898259738609 disk.drbd1260.out‚0.765625) disk.nvme0c0n1.out‚128572.56103515625 disk.sda.in‚0 disk.sda.out‚ 168.11328125 ”ø½‚(:’ “ïçÔ·Ñûx gpu.0.gpu‚89.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚276.577' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.315400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–‚ût¾‚(:é “ïçÔ÷û«y% disk./.usageGB‚1159.2846221923828! disk.nvme0c0n1.in‚ 14975.40625 disk.sda.in‚0 disk.sda.out‚ 168.484375 disk.sda3.out‚ 168.484375 disk.drbd1260.out‚ 0.76953125 cpu‚25.067246088741882" disk.dm-0.out‚129028.615234375 disk.drbd1260.in‚0% disk.nvme0n1.out‚129028.615234375 disk.sda1.in‚0$ memory_percent‚89.61727619171143) proc.memory.availableMB‚ 1701.10546875" proc.memory.rssMB‚ 3144.8203125 network.sent‚ 7.944397e+06' disk.nvme0c0n1.out‚129028.615234375 disk.dm-1.out‚ 168.484375 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.755407e+06) disk./.usagePercent‚77.80056369991756 disk.dm-0.in‚ 14975.40625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 14975.40625 disk.sda2.in‚0* proc.memory.percent‚19.194459915161133æa/’¿‚(:‡ ¢ïçÔ•w gpu.0.gpu‚59.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚232.691' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.5382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f2Úö€À‚(:õ ¢ïçÔº£ßw disk.sda1.out‚0 network.recv‚ 1.758983e+06% disk./.usageGB‚1159.2847290039062# disk.nvme0c0n1.in‚ 15093.8203125 disk.dm-0.in‚ 15093.8203125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 169.015625 disk.drbd1260.out‚ 0.76953125 network.sent‚ 7.949238e+06' disk.nvme0n1.out‚129221.31005859375 disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1701.10546875" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133 cpu‚25.067304036342442$ disk.dm-0.out‚129221.31005859375 disk.sda.out‚ 169.015625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 15093.8203125 disk.sda2.in‚0$ memory_percent‚89.61727619171143 proc.cpu.threads‚17) disk./.usagePercent‚77.80057086812852) disk.nvme0c0n1.out‚129221.31005859375 disk.dm-1.in‚0 disk.dm-1.out‚ 169.015625PrŒ2Á- _wandbruntime‚5509  _runtime‚5509ÈÕ[ÆÂÀ _runtime‚ 5509.19901525 _step‚250 & lr-AdamW/pg1‚0.0006045183215937762 & lr-AdamW/pg2‚0.0006045183215937762 trainer/global_step‚3099 % _timestamp‚1.7884753024431431e+09ëvƒ3ËÃÅ % _timestamp‚1.7884753024431431e+09 _runtime‚ 5509.19901525 _step‚250 & lr-AdamW/pg1‚0.0006045183215937762 & lr-AdamW/pg2‚0.0006045183215937762 trainer/global_step‚3099úÌë'Û2Ä- _wandbruntime‚5509  _runtime‚5509ã_²(ÅüO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443843708990464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77485305 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65285655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 trainer/global_step‚3099 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41252222 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚486199767983003 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚490320294685696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7540584012288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142770966 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65285661 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77458966 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77458966 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69441320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10386784311296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 % _timestamp‚1.7884753024445853e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚478672394604304 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65285655 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7540584012288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497864489358336 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚486203402073651 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚478672420032272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77485305 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7527373378699 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28189098 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7544194672640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490320294685696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚5509.200685489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497860852306432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28189816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142744621 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚490320268294144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454230442098688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10386858347008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443843657787392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497864489358336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚718 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454230567337472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7530982041379 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7544194672640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497860852306432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912­UU!(ÆP PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65285661 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77458966 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77458966 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69441320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10386784311296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 % _timestamp‚1.7884753024445853e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚478672394604304 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65285655 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7540584012288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497864489358336 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚486203402073651 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚478672420032272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77485305 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7527373378699 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28189098 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7544194672640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490320294685696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚5509.200685489 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497860852306432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28189816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142744621 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚490320268294144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454230442098688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10386858347008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443843657787392 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚497864489358336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚718 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚454230567337472 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7530982041379 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7544194672640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497860852306432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443843708990464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77485305 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65285655 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 trainer/global_step‚3099 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41252222 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚486199767983003 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚490320294685696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7540584012288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142770966 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0û7{–2Ç- _wandbruntime‚5510  _runtime‚5510ìß$ëȆN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142815983 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65306085 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69462987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7529779681985 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490474072244736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490474045853184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490474045853184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚490474072244736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7542994493952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7542994493952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10390177096192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3099 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69463704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41265027 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚478822749273408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77483533 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65306079 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 % _timestamp‚1.7884753035323045e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142815983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28197960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443982948176384 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 _runtime‚5510.289094326 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77509898 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚498020677412352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10390103073792 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 % _timestamp‚1.7884753035323045e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142815983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28197960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443982948176384 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 _runtime‚5510.289094326 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77509898 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚498020677412352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10390103073792 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142815983 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65306085 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69462987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7529779681985 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚490474072244736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490474045853184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490474045853184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚490474072244736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7542994493952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7542994493952 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10390177096192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3099 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69463704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41265027 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚478822749273408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77483533 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719üµSè2Ê- _wandbruntime‚5510  _runtime‚5510Z=ÆUZËÔ 1loss/FBt1/assignment_loss‚0.01076864274800755 0loss/FBt4/assignment_loss‚0.8797781780362129 %loss/total_loss‚51.09211251093426 1loss/FBt2/assignment_loss‚0.29877123510823944 /loss/FRt2/detection_loss‚2.0971831216894348 1loss/SRqqt1/detection_loss‚1.1250245353189585 % _timestamp‚1.7884753035346332e+09 2loss/SRqqt1/assignment_loss‚1.4114588923299702 0loss/FBt3/assignment_loss‚0.6277850586200311 1loss/SRqqt4/detection_loss‚2.4529310510708737 epoch‚2 2loss/SRqqt4/assignment_loss‚1.6001937824946184 0loss/SRqqt3/detection_loss‚2.608849094382354 /loss/FBt4/detection_loss‚3.2404405653476713 _step‚253 0loss/FRt4/assignment_loss‚3.6098432478151823 2loss/SRqqt3/assignment_loss‚1.9005403657044684 .loss/FBt3/detection_loss‚2.243580557051159 2loss/SRqqt2/assignment_loss‚1.6827193780949241 /loss/FRt1/detection_loss‚1.8695100754414296 /loss/FBt1/detection_loss‚3.2315860033035277 /loss/FRt2/assignment_loss‚3.025678962469101 .loss/FRt3/assignment_loss‚4.97371518611908 /loss/FRt3/detection_loss‚3.7872510657590976 1loss/SRqqt2/detection_loss‚2.4881482532149866 trainer/global_step‚3099 .loss/FRt1/assignment_loss‚2.37985693963606 /loss/FRt4/detection_loss‚2.8256501429959346 /loss/FBt2/detection_loss‚0.7208481761829361 _runtime‚5510.290889641QÑŸ'_ÌÙ 1loss/SRqqt1/detection_loss‚1.1250245353189585 % _timestamp‚1.7884753035346332e+09 2loss/SRqqt1/assignment_loss‚1.4114588923299702 0loss/FBt3/assignment_loss‚0.6277850586200311 1loss/SRqqt4/detection_loss‚2.4529310510708737 epoch‚2 2loss/SRqqt4/assignment_loss‚1.6001937824946184 0loss/SRqqt3/detection_loss‚2.608849094382354 /loss/FBt4/detection_loss‚3.2404405653476713 _step‚253 0loss/FRt4/assignment_loss‚3.6098432478151823 2loss/SRqqt3/assignment_loss‚1.9005403657044684 .loss/FBt3/detection_loss‚2.243580557051159 2loss/SRqqt2/assignment_loss‚1.6827193780949241 /loss/FRt1/detection_loss‚1.8695100754414296 /loss/FBt1/detection_loss‚3.2315860033035277 /loss/FRt2/assignment_loss‚3.025678962469101 .loss/FRt3/assignment_loss‚4.97371518611908 /loss/FRt3/detection_loss‚3.7872510657590976 1loss/SRqqt2/detection_loss‚2.4881482532149866 trainer/global_step‚3099 .loss/FRt1/assignment_loss‚2.37985693963606 /loss/FRt4/detection_loss‚2.8256501429959346 /loss/FBt2/detection_loss‚0.7208481761829361 _runtime‚5510.290889641 1loss/FBt1/assignment_loss‚0.01076864274800755 0loss/FBt4/assignment_loss‚0.8797781780362129 %loss/total_loss‚51.09211251093426 1loss/FBt2/assignment_loss‚0.29877123510823944 /loss/FRt2/detection_loss‚2.0971831216894348ý¿ú`wÍ‚(:ì ±ïçÔž½˜x* proc.memory.percent‚19.194459915161133) disk./.usagePercent‚77.80056600398537% disk./.usageGB‚1159.2846565246582 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 15093.8203125 cpu‚25.06743177342856 network.recv‚ 1.76343e+06 disk.sda.out‚ 169.34375! disk.nvme0n1.in‚ 15093.8203125) proc.memory.availableMB‚ 1700.82421875 proc.cpu.threads‚17 network.sent‚ 7.955407e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7734375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 15093.8203125( disk.nvme0c0n1.out‚129319.4033203125# disk.dm-0.out‚129319.4033203125" proc.memory.rssMB‚ 3144.8203125 disk.dm-1.in‚0 disk.dm-1.out‚ 169.34375 disk.sda3.in‚0 disk.sda3.out‚ 169.34375& disk.nvme0n1.out‚129319.4033203125 disk.sda2.in‚0$ memory_percent‚89.61899280548096Ú6ø΂(:„ ±ïçÔéÒÃy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Ù~Ï‚(:ó ÀïçÔä‹‹z disk.sda1.in‚0# disk.nvme0c0n1.in‚ 15093.8203125 disk.dm-1.out‚ 169.68359375$ memory_percent‚89.61756229400635* proc.memory.percent‚19.194459915161133% disk./.usageGB‚1159.2654609680176 disk.drbd1260.out‚ 0.7734375 disk.sda1.out‚0' disk.nvme0c0n1.out‚129537.408203125 disk.dm-0.in‚ 15093.8203125 disk.dm-1.in‚0" proc.memory.rssMB‚ 3144.8203125 cpu‚25.068167111178074 network.recv‚ 1.76869e+06 disk.sda.in‚0% disk.nvme0n1.out‚129537.408203125 proc.cpu.threads‚17) disk./.usagePercent‚77.79927777407306 disk.sda.out‚ 169.68359375 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 15093.8203125 disk.sda2.in‚0" disk.dm-0.out‚129537.408203125) proc.memory.availableMB‚ 1701.05859375 network.sent‚ 8.064764e+06 disk.sda3.out‚ 169.68359375 disk.sda2.out‚0|@ËÇЂ(:… ÀïçÔ×â›{ gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.278' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6556 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òfâ÷Ñ‚(:… ÏïçÔ¶“Åx gpu.0.gpu‚57.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚138.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.742 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þtt»€Ò‚(:õ ÏïçÔ«Í©y disk.dm-1.in‚0 disk.dm-1.out‚ 170.03515625 disk.sda.in‚0 disk.sda.out‚ 170.03515625 disk.sda3.out‚ 170.03515625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.77734375 network.recv‚ 1.773305e+06 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚129924.4912109375! disk.nvme0n1.in‚ 15093.8203125& disk.nvme0n1.out‚129924.4912109375$ memory_percent‚89.61749076843262 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 15093.8203125( disk.nvme0c0n1.out‚129924.4912109375 disk.dm-0.in‚ 15093.8203125( proc.memory.availableMB‚ 1701.0703125" proc.memory.rssMB‚ 3144.8203125 cpu‚25.06861269287956 network.sent‚ 8.069394e+06) disk./.usagePercent‚77.79255476021638 disk.sda3.in‚0* proc.memory.percent‚19.194459915161133 proc.cpu.threads‚17$ disk./.usageGB‚1159.165283203125’(“dvÓ‚(:ë ÞïçÔÌÑøx disk.sda3.out‚ 170.65625 disk.drbd1260.out‚ 0.77734375& disk.nvme0n1.out‚130437.3623046875 network.recv‚ 1.776648e+06 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 170.65625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 8.074124e+06% disk./.usageGB‚1159.2439727783203 disk.nvme0n1.in‚ 15229.078125 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚89.61749076843262" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133) disk./.usagePercent‚77.79783568363268" disk.nvme0c0n1.in‚ 15229.078125 disk.dm-0.in‚ 15229.078125( proc.memory.availableMB‚ 1701.0703125 cpu‚25.068699678055665 proc.cpu.threads‚17( disk.nvme0c0n1.out‚130437.3623046875# disk.dm-0.out‚130437.3623046875 disk.dm-1.out‚ 170.65625^]5’Ô‚(:‡ ÞïçÔú˜Äy gpu.0.gpu‚65.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚117.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìMŸßyÕ‚(:î íïçÔ¼¤¯x( disk.nvme0c0n1.out‚130782.9130859375 disk.drbd1260.out‚0.78125 disk.nvme0n1.in‚ 15229.078125 disk.sda2.in‚0" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133 cpu‚25.06883145801613 disk.dm-1.in‚0 disk.sda.in‚0 network.sent‚ 8.080292e+06% disk./.usageGB‚1159.3461875915527 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 15229.078125 disk.dm-0.in‚ 15229.078125# disk.dm-0.out‚130782.9130859375 disk.sda.out‚ 171.0703125 disk.sda3.out‚ 171.0703125& disk.nvme0n1.out‚130782.9130859375$ memory_percent‚89.61904048919678 proc.cpu.threads‚17 disk.dm-1.out‚ 171.0703125 network.recv‚ 1.781545e+06) disk./.usagePercent‚77.80469540551266 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1700.81640625#­KÖ‚(:… íïçÔÏ·¥z gpu.0.gpu‚12.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚191.795' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.359 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä/ëñ2×- _wandbruntime‚5591  _runtime‚5591µÈ&BÅØ¿ _step‚254 % lr-AdamW/pg1‚0.000602047401346983 % lr-AdamW/pg2‚0.000602047401346983 trainer/global_step‚3149 % _timestamp‚1.7884753844427583e+09 _runtime‚5591.199550354æ@lÊÙÄ trainer/global_step‚3149 % _timestamp‚1.7884753844427583e+09 _runtime‚5591.199550354 _step‚254 % lr-AdamW/pg1‚0.000602047401346983 % lr-AdamW/pg2‚0.000602047401346983þfÛ§2Ú- _wandbruntime‚5591  _runtime‚5591Ši! /ÛüO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41892472 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10552488296448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78717007 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78690668 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450808177236992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144997523 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7651310796887 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66306861 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884753844466884e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28631081 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚486188882089072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505670268755456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498009172637696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28631823 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚493836558795311 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _runtime‚5591.203833392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498009172637696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70523553 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461360665533440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7661122509312 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66306861 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884753844466884e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28631081 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚486188882089072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505670268755456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498009172637696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28631823 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚493836558795311 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _runtime‚5591.203833392 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498009172637696 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70523553 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461360665533440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7661122509312 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚731 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450947518829056 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145068885 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66327285 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28639880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498162923805184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7667143664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66327285 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10555871943680 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5593.249552884 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461503390772736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7653717100381 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66327279 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70545888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78741600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498162950196736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10555797921280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78715235 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505830093861376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70545157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66327279 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7650108437493 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 trainer/global_step‚3149 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145042514 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚486339211330208 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41905277 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚486339185902240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145042514 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚493989294339733 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505830093861376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552´³Ú…Å$ß‹N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145042514 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚493989294339733 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505830093861376 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7663532990976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚256 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚731 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450947518829056 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145068885 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66327285 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28639880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498162923805184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7667143664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66327285 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10555871943680 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5593.249552884 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461503390772736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7653717100381 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66327279 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70545888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78741600 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498162950196736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10555797921280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78715235 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505830093861376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70545157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66327279 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7650108437493 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 trainer/global_step‚3149 H:?‰©»LDeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145042514 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚486339211330208 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41905277 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚486339185902240€y›ð½2à- _wandbruntime‚5593  _runtime‚55938¿·UáÏ 1loss/SRqqt1/assignment_loss‚1.478554350190929 1loss/SRqqt3/assignment_loss‚1.349566181068835 0loss/SRqqt1/detection_loss‚1.017408741253709 epoch‚2 /loss/FRt3/assignment_loss‚5.239953362941742 2loss/SRqqt2/assignment_loss‚1.9132990936438243 /loss/FRt3/detection_loss‚3.4126324236392973 1loss/SRqqt4/detection_loss‚2.3801311602195105 .loss/FBt1/detection_loss‚3.477148473262787 trainer/global_step‚3149 0loss/FRt1/assignment_loss‚2.2009948440220044 1loss/FBt4/assignment_loss‚0.27247626283497084 /loss/FRt1/detection_loss‚1.6404994378192013 0loss/FBt1/assignment_loss‚0.4439541461567084 1loss/SRqqt2/detection_loss‚2.3917617897192636 _step‚257 /loss/FRt2/detection_loss‚2.4861839681863787 /loss/FRt2/assignment_loss‚2.924145959317684 1loss/SRqqt3/detection_loss‚2.7053907047147336 /loss/FBt2/detection_loss‚0.9089389156189657 %loss/total_loss‚51.04496067350792 .loss/FRt4/assignment_loss‚4.16376682387458 0loss/FBt2/assignment_loss‚0.2699011860654995 .loss/FRt4/detection_loss‚2.503303305308024 .loss/FBt3/detection_loss‚2.140738120204524 /loss/FBt4/detection_loss‚2.6915142724388526 2loss/SRqqt4/assignment_loss‚2.4553069637881384 0loss/FBt3/assignment_loss‚0.5773901872177559 $ _timestamp‚1.788475386497152e+09 _runtime‚5593.253554009e‹–rZâÔ 2loss/SRqqt4/assignment_loss‚2.4553069637881384 0loss/FBt3/assignment_loss‚0.5773901872177559 $ _timestamp‚1.788475386497152e+09 _runtime‚5593.253554009 1loss/SRqqt1/assignment_loss‚1.478554350190929 1loss/SRqqt3/assignment_loss‚1.349566181068835 0loss/SRqqt1/detection_loss‚1.017408741253709 epoch‚2 /loss/FRt3/assignment_loss‚5.239953362941742 2loss/SRqqt2/assignment_loss‚1.9132990936438243 /loss/FRt3/detection_loss‚3.4126324236392973 1loss/SRqqt4/detection_loss‚2.3801311602195105 .loss/FBt1/detection_loss‚3.477148473262787 trainer/global_step‚3149 0loss/FRt1/assignment_loss‚2.2009948440220044 1loss/FBt4/assignment_loss‚0.27247626283497084 /loss/FRt1/detection_loss‚1.6404994378192013 0loss/FBt1/assignment_loss‚0.4439541461567084 1loss/SRqqt2/detection_loss‚2.3917617897192636 _step‚257 /loss/FRt2/detection_loss‚2.4861839681863787 /loss/FRt2/assignment_loss‚2.924145959317684 1loss/SRqqt3/detection_loss‚2.7053907047147336 /loss/FBt2/detection_loss‚0.9089389156189657 %loss/total_loss‚51.04496067350792 .loss/FRt4/assignment_loss‚4.16376682387458 0loss/FBt2/assignment_loss‚0.2699011860654995 .loss/FRt4/detection_loss‚2.503303305308024 .loss/FBt3/detection_loss‚2.140738120204524 /loss/FBt4/detection_loss‚2.6915142724388526ëh!åpã‚(:å üïçÔÌïÏx disk.nvme0n1.in‚ 15229.078125 disk.sda2.in‚0& proc.memory.availableMB‚ 1700.53125" proc.memory.rssMB‚ 3144.8203125) disk./.usagePercent‚77.80471486208526$ disk./.usageGB‚1159.346477508545 disk.dm-1.in‚0 disk.sda.out‚ 171.46875$ memory_percent‚89.62078094482422 proc.cpu.threads‚17 network.sent‚ 8.085227e+06 disk.sda3.out‚ 171.46875& disk.nvme0n1.out‚130975.6904296875 disk.sda1.out‚0* proc.memory.percent‚19.194459915161133 cpu‚25.069438547106117" disk.nvme0c0n1.in‚ 15229.078125( disk.nvme0c0n1.out‚130975.6904296875# disk.dm-0.out‚130975.6904296875 disk.dm-1.out‚ 171.46875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚ 1.784769e+06 disk.dm-0.in‚ 15229.078125 disk.sda.in‚0 disk.drbd1260.out‚0.78125áôU¥œä‚(:‘ üïçÔÒééx gpu.0.gpu‚71.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚139.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.816000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕŒ¬å‚(:„ ‹ðçÔõ‘{ gpu.0.gpu‚20.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚142.34' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ‰ö%qæ‚(:æ ‹ðçԞ犇) disk.nvme0c0n1.out‚131149.87255859375 disk.dm-0.in‚ 15229.078125$ disk.dm-0.out‚131149.87255859375 disk.dm-1.in‚0 disk.sda.out‚171.9375 disk.sda3.out‚171.9375 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 15229.078125 network.sent‚ 8.194119e+06( disk./.usagePercent‚77.8050307753825 disk.sda3.in‚0' disk.nvme0n1.out‚131149.87255859375 disk.sda1.out‚0$ memory_percent‚89.61944580078125" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133 network.recv‚ 1.791559e+06% disk./.usageGB‚1159.3511848449707 disk.dm-1.out‚171.9375 disk.sda.in‚0 disk.drbd1260.out‚ 0.78515625 disk.sda2.in‚0 disk.sda1.in‚0# proc.memory.availableMB‚1700.75" disk.nvme0c0n1.in‚ 15229.078125 disk.sda2.out‚0 cpu‚25.069518590974482 proc.cpu.threads‚17€Žÿ€‘ç‚(:† šðçÔ’•Úx gpu.0.gpu‚17.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.563' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9126 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cø2nè‚(:ã šðçÔèÈ×y network.sent‚ 8.199136e+06 network.recv‚ 1.795001e+06% disk./.usageGB‚1159.3276977539062 disk.nvme0n1.in‚ 15296.625) disk.nvme0c0n1.out‚131727.18505859375$ disk.dm-0.out‚131727.18505859375 disk.dm-1.in‚0 disk.dm-1.out‚ 172.35546875 disk.drbd1260.in‚0' disk.nvme0n1.out‚131727.18505859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 15296.625 disk.sda.in‚0" memory_percent‚89.617919921875 proc.memory.availableMB‚1701( disk./.usagePercent‚77.8034545369942 disk.drbd1260.out‚ 0.78515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 15296.625" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133 cpu‚25.069370582445252 disk.sda.out‚ 172.35546875 disk.sda3.in‚0 disk.sda3.out‚ 172.35546875 proc.cpu.threads‚17¥vE‘é‚(:† ©ðçÔõίy gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.127' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.6254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h<5{ê‚(:ð ©ðçÔ鱌{ disk.sda2.out‚0$ memory_percent‚89.61942195892334* proc.memory.percent‚19.194459915161133 network.sent‚ 8.20708e+06 network.recv‚ 1.799503e+06) disk./.usagePercent‚77.80765050048039 disk.dm-1.in‚0 disk.sda3.out‚ 172.953125 cpu‚25.069786545925375% disk./.usageGB‚1159.3902206420898 disk.drbd1260.in‚0% disk.nvme0n1.out‚132015.021484375$ disk.nvme0c0n1.in‚15385.08984375' disk.nvme0c0n1.out‚132015.021484375" disk.dm-0.out‚132015.021484375) proc.memory.availableMB‚ 1700.75390625" proc.memory.rssMB‚ 3144.8203125 disk.sda.out‚ 172.953125 disk.drbd1260.out‚ 0.7890625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚15385.08984375 proc.cpu.threads‚17 disk.dm-1.out‚ 172.953125 disk.sda.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚15385.08984375Îáë‚(:ö ¸ðçÔæ­Ãw disk.dm-0.in‚15385.08984375 disk.dm-1.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚15385.08984375% disk.nvme0n1.out‚132265.412109375 network.sent‚ 8.210002e+06" disk.dm-0.out‚132265.412109375 disk.dm-1.out‚ 173.30859375 disk.sda.out‚ 173.30859375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7890625 disk.sda2.in‚0$ memory_percent‚89.61939811706543 network.recv‚ 1.802801e+06$ disk.nvme0c0n1.in‚15385.08984375 disk.sda3.out‚ 173.30859375 disk.sda2.out‚0 disk.sda1.out‚0 cpu‚25.070332355756385 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1700.7578125" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133) disk./.usagePercent‚77.80920753830364% disk./.usageGB‚1159.4134216308594' disk.nvme0c0n1.out‚132265.412109375¦bä‘ì‚(:† ¸ðçÔ¥ÎÊx gpu.0.gpu‚51.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚131.649' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uѪ…í‚(:ú ÇðçÔªøðx proc.cpu.threads‚17 network.sent‚ 8.216543e+06) disk./.usagePercent‚77.81340222175217% disk./.usageGB‚1159.4759254455566 disk.dm-0.in‚15385.08984375 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd1260.out‚ 0.79296875" disk.nvme0n1.in‚15385.08984375 network.recv‚ 1.807732e+06$ disk.nvme0c0n1.in‚15385.08984375# disk.dm-0.out‚132564.3642578125 disk.dm-1.out‚ 173.77734375 disk.sda3.out‚ 173.77734375 disk.drbd1260.in‚0& disk.nvme0n1.out‚132564.3642578125 disk.sda2.out‚0 disk.sda.out‚ 173.77734375 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1700.7578125" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133 cpu‚25.070830842318728( disk.nvme0c0n1.out‚132564.3642578125 disk.sda1.out‚0$ memory_percent‚89.61939811706543¥e¸î‚(:’ ÇðçÔÆ€‘y gpu.0.gpu‚82.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.214' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.842800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&¿µ…‘ï‚(:† ÖðçÔéÊ®w gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚204.605' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.921 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÎb…ð‚(:ú ÖðçÔä°Îx disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚19.194459915161133 network.sent‚ 8.223326e+06 network.recv‚ 1.811122e+06% disk./.usageGB‚1159.5774574279785 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚132789.7939453125# disk.dm-0.out‚132789.7939453125 disk.dm-1.out‚ 174.15234375) disk./.usagePercent‚77.82021611828351 disk.drbd1260.out‚ 0.79296875 disk.dm-0.in‚15385.08984375 disk.sda.in‚0 disk.sda.out‚ 174.15234375$ memory_percent‚89.62092399597168( proc.memory.availableMB‚ 1700.5078125" proc.memory.rssMB‚ 3144.8203125 disk.sda3.in‚0 disk.sda3.out‚ 174.15234375" disk.nvme0n1.in‚15385.08984375$ disk.nvme0c0n1.in‚15385.08984375 cpu‚25.071341311618156 proc.cpu.threads‚17 disk.drbd1260.in‚0& disk.nvme0n1.out‚132789.7939453125 disk.sda1.in‚0ñ€( 2ñ- _wandbruntime‚5689  _runtime‚5689)–bÇòÁ & lr-AdamW/pg1‚0.0005995189378243544 & lr-AdamW/pg2‚0.0005995189378243544 trainer/global_step‚3199 % _timestamp‚1.7884754830174189e+09 _runtime‚5689.773407056 _step‚258UU¹qÌóÆ _runtime‚5689.773407056 _step‚258 & lr-AdamW/pg1‚0.0005995189378243544 & lr-AdamW/pg2‚0.0005995189378243544 trainer/global_step‚3199 % _timestamp‚1.7884754830174189e+09‚mƒ2ô- _wandbruntime‚5689  _runtime‚5689%'šÞ(õúO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513479685204480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67328055 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7768030889715 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚505698050589696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7781661006336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7781661006336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7771639552395 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 _step‚259 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚505698024198144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71606368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147276770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71605636 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42532727 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10718807647232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10718733611520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513479685204480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513483322256384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7785271666688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 trainer/global_step‚3199 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493705318717904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67328061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468491430298112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7785271666688 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42532722 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468491555536896 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79922370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29073641 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79948709 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79922370 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚505698024198144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79948709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29072914 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67328061 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚505698050589696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67328055 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5134833222563841¼Í¹Ó$öÿO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29072914 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67328061 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚505698050589696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67328055 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚513483322256384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513479685204480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67328055 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7768030889715 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚505698050589696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7781661006336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7781661006336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7771639552395 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 _step‚259 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚505698024198144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71606368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147276770 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71605636 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42532727 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10718807647232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10718733611520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513479685204480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513483322256384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7785271666688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 trainer/global_step‚3199 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493705318717904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67328061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚468491430298112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7785271666688 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42532722 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚468491555536896 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79922370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29073641 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.smauÑ2all_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79948709 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79922370 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚505698024198144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79948709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187ƒ–‰»22÷- _wandbruntime‚5690  _runtime‚5690ôôYõ 'ø†N WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚501629719242897 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚493855673387008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7784071488000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468634020354048 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79946937 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513635873245184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10722107300864 _step‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457911987075584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67348485 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚505851801757184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79973302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 % _timestamp‚1.7884754841314223e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147321787 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71627289 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29082466 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147295416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7770437193001 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7784071488000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _runtime‚5690.888296855 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚505851828148736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67348479 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29081762 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71627998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3199 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79973302 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147295416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42545527 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42545532 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505851801757184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7787682161664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468634145579520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7774045855889 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513639510310400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67348485 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147321787 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚505851828148736 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400ÚUuR'ù‹N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚505851828148736 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚501629719242897 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚493855673387008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7784071488000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚468634020354048 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79946937 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513635873245184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚709 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10722107300864 _step‚260 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457911987075584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67348485 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚505851801757184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79973302 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 % _timestamp‚1.7884754841314223e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147321787 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71627289 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29082466 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147295416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7770437193001 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7784071488000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _runtime‚5690.888296855 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚505851828148736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67348479 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29081762 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71627998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3199 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79973302 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147295416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42545527 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42545532 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505851801757184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7787682161664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468634145579520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7774045855889 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513639510310400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67348485 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147321787 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424„}÷L2ú- _wandbruntime‚5690  _runtime‚5690€ü¥ÎZûÔ 1loss/FBt1/assignment_loss‚0.19524158178440607 /loss/FRt2/detection_loss‚2.1245090685163937 %loss/total_loss‚51.39613326520227 1loss/SRqqt2/assignment_loss‚1.955682076215744 trainer/global_step‚3199 % _timestamp‚1.7884754841339796e+09 0loss/FRt2/assignment_loss‚2.6211023603876433 1loss/SRqqt1/detection_loss‚1.0170867135318427 0loss/FBt4/assignment_loss‚0.3110367897897959 /loss/FRt3/detection_loss‚3.5871687531471252 0loss/SRqqt2/detection_loss‚2.835506093502045 .loss/FBt1/detection_loss‚3.391766279935837 _step‚261 0loss/FRt3/assignment_loss‚4.1122417317496405 2loss/SRqqt1/assignment_loss‚1.4146402705030945 2loss/SRqqt4/assignment_loss‚2.0369928375338064 /loss/FRt4/detection_loss‚2.9743334343939116 1loss/SRqqt4/detection_loss‚2.4630614166909996 .loss/FBt2/detection_loss‚0.764404768874841 /loss/FBt3/detection_loss‚2.0436931972702346 0loss/FRt1/assignment_loss‚2.7842688540617626 2loss/SRqqt3/assignment_loss‚2.1312705084033636 1loss/SRqqt3/detection_loss‚3.1171254930288894 /loss/FBt4/detection_loss‚2.3160632683171167 /loss/FRt4/assignment_loss‚4.488286784200957 0loss/FBt2/assignment_loss‚0.3356589890960776 0loss/FBt3/assignment_loss‚0.3704922142654444 .loss/FRt1/detection_loss‚2.004499780001303 _runtime‚5690.889395577 epoch‚2\¦p¯_üÙ 1loss/SRqqt1/detection_loss‚1.0170867135318427 0loss/FBt4/assignment_loss‚0.3110367897897959 /loss/FRt3/detection_loss‚3.5871687531471252 0loss/SRqqt2/detection_loss‚2.835506093502045 .loss/FBt1/detection_loss‚3.391766279935837 _step‚261 0loss/FRt3/assignment_loss‚4.1122417317496405 2loss/SRqqt1/assignment_loss‚1.4146402705030945 2loss/SRqqt4/assignment_loss‚2.0369928375338064 /loss/FRt4/detection_loss‚2.9743334343939116 1loss/SRqqt4/detection_loss‚2.4630614166909996 .loss/FBt2/detection_loss‚0.764404768874841 /loss/FBt3/detection_loss‚2.0436931972702346 0loss/FRt1/assignment_loss‚2.7842688540617626 2loss/SRqqt3/assignment_loss‚2.1312705084033636 1loss/SRqqt3/detection_loss‚3.1171254930288894 /loss/FBt4/detection_loss‚2.3160632683171167 /loss/FRt4/assignment_loss‚4.488286784200957 0loss/FBt2/assignment_loss‚0.3356589890960776 0loss/FBt3/assignment_loss‚0.3704922142654444 .loss/FRt1/detection_loss‚2.004499780001303 _runtime‚5690.889395577 epoch‚2 1loss/FBt1/assignment_loss‚0.19524158178440607 /loss/FRt2/detection_loss‚2.1245090685163937 %loss/total_loss‚51.39613326520227 1loss/SRqqt2/assignment_loss‚1.955682076215744 trainer/global_step‚3199 % _timestamp‚1.7884754841339796e+09 0loss/FRt2/assignment_loss‚2.6211023603876433…SžQ‰ý‚(:þ åðçÔÛìÚx disk.sda1.in‚0" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133 disk.sda3.out‚ 174.56640625 disk.drbd1260.out‚ 0.80078125 network.recv‚ 1.815333e+06$ disk.dm-0.out‚133377.06201171875 disk.dm-1.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚133377.06201171875) proc.memory.availableMB‚ 1700.50390625 network.sent‚ 8.227529e+06% disk./.usageGB‚1159.5777053833008 disk.sda.in‚0" disk.nvme0n1.in‚15539.37109375 disk.sda1.out‚0$ disk.nvme0c0n1.in‚15539.37109375 cpu‚25.071802870781266 proc.cpu.threads‚17) disk./.usagePercent‚77.82023275877323 disk.dm-0.in‚15539.37109375 disk.dm-1.out‚ 174.56640625 disk.sda.out‚ 174.56640625 disk.sda3.in‚0' disk.nvme0n1.out‚133377.06201171875 disk.sda2.out‚0$ memory_percent‚89.62094783782959 disk.drbd1260.in‚0¶ÛÕ=þ‚(:’ åðçÔÉé‚z gpu.0.gpu‚58.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚178.636' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.727199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n—Øc’ÿ‚(:‡ ôðçÔ«ÿÃy gpu.0.gpu‚54.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚187.962' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.5924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­€¿j‡€‚(:ü ôðçÔÒÉŽ{) disk.nvme0c0n1.out‚133688.62451171875 disk.dm-1.in‚0* proc.memory.percent‚19.194459915161133 network.sent‚ 8.334238e+06 disk.sda.out‚ 175.22265625 disk.drbd1260.out‚ 0.80078125$ disk.nvme0c0n1.in‚15539.37109375$ memory_percent‚89.62101936340332 cpu‚25.072370612866685 proc.cpu.threads‚17% disk./.usageGB‚1159.8511924743652 disk.sda.in‚0 disk.sda3.out‚ 175.22265625 disk.drbd1260.in‚0" disk.nvme0n1.in‚15539.37109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚15539.37109375 network.recv‚ 1.82043e+06) disk./.usagePercent‚77.83858670692553 disk.sda2.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚133688.62451171875 disk.dm-1.out‚ 175.22265625( proc.memory.availableMB‚ 1700.4921875" proc.memory.rssMB‚ 3144.8203125 disk.sda3.in‚0' disk.nvme0n1.out‚133688.624511718752Vn$„‚(:ù ƒñçÔ²Óx disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 175.6796875 proc.cpu.threads‚17$ disk./.usageGB‚1159.851173400879 disk.sda1.out‚0$ disk.nvme0c0n1.in‚15539.37109375$ disk.dm-0.out‚133892.12158203125 disk.dm-1.out‚ 175.6796875 disk.sda.in‚0) proc.memory.availableMB‚ 1700.48828125* proc.memory.percent‚19.194459915161133 disk.drbd1260.out‚ 0.8046875 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚15539.37109375 disk.sda.out‚ 175.6796875" proc.memory.rssMB‚ 3144.8203125 network.recv‚ 1.825326e+06 disk.drbd1260.in‚0' disk.nvme0n1.out‚133892.62158203125) disk.nvme0c0n1.out‚133892.37158203125 disk.dm-1.in‚0$ memory_percent‚89.62104320526123 cpu‚25.072849834324753 network.sent‚ 8.340706e+06) disk./.usagePercent‚77.83858542688785" disk.nvme0n1.in‚15539.37109375*¬…Çœ‚‚(:‘ ƒñçÔóæ˜y gpu.0.gpu‚17.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚127.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.568199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê0”"…ƒ‚(:ú ’ñçÔÉœx network.recv‚ 1.828652e+06) disk./.usagePercent‚77.83858542688785 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 176.05859375 disk.drbd1260.out‚ 0.8046875$ memory_percent‚89.62106704711914 network.sent‚ 8.345559e+06 disk.sda1.out‚0$ disk.nvme0c0n1.in‚15539.37109375 disk.drbd1260.in‚0' proc.memory.availableMB‚ 1700.484375 cpu‚25.073352218261412 proc.cpu.threads‚17 disk.sda2.out‚0$ disk.dm-0.out‚134190.83251953125 disk.dm-1.in‚0" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133$ disk./.usageGB‚1159.851173400879" disk.nvme0n1.in‚15539.37109375' disk.nvme0n1.out‚134190.83251953125 disk.sda1.in‚0) disk.nvme0c0n1.out‚134190.83251953125 disk.dm-0.in‚15539.37109375 disk.dm-1.out‚ 176.05859375 disk.sda.out‚ 176.05859375¢`þ’„‚(:‡ ’ñçÔ€¹Óx gpu.0.gpu‚57.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø +€…‚(:õ ¡ñçÔŽ©þx disk.dm-1.out‚ 176.46484375 disk.sda3.in‚0 disk.sda3.out‚ 176.46484375 disk.drbd1260.in‚0 disk.sda1.out‚0 cpu‚25.07392506183219 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 176.46484375& disk.nvme0n1.out‚134536.7255859375& proc.memory.availableMB‚ 1700.78125( disk./.usagePercent‚77.8370104685372# disk.dm-0.out‚134536.7255859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚134536.7255859375" proc.memory.rssMB‚ 3144.8203125* proc.memory.percent‚19.194459915161133 network.sent‚ 8.35182e+06 network.recv‚ 1.833298e+06% disk./.usageGB‚1159.8277053833008 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.80859375" disk.nvme0n1.in‚15661.28515625$ disk.nvme0c0n1.in‚15661.28515625$ memory_percent‚89.61925506591797 disk.dm-0.in‚15661.28515625KÊfN†‚(:’ ¡ñçÔè‘«z gpu.0.gpu‚55.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚115.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.131200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ð¾L„‡‚(:ù °ñçÔ­õ¹x" disk.nvme0n1.in‚15661.28515625 disk.sda2.out‚0( disk.nvme0c0n1.out‚134931.3779296875# disk.dm-0.out‚134931.3779296875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 cpu‚25.074422269009037 network.recv‚ 1.836759e+06) disk./.usagePercent‚77.87068134746309 disk.dm-1.out‚ 176.80078125& disk.nvme0n1.out‚134931.3779296875 disk.sda2.in‚0$ memory_percent‚89.62070941925049" proc.memory.rssMB‚ 3144.8203125 disk.drbd1260.out‚ 0.80859375 disk.sda1.in‚0) proc.memory.availableMB‚ 1700.54296875* proc.memory.percent‚19.194459915161133 proc.cpu.threads‚17 network.sent‚ 8.3566e+06% disk./.usageGB‚1160.3294258117676$ disk.nvme0c0n1.in‚15661.28515625 disk.dm-0.in‚15661.28515625 disk.sda.out‚ 176.80078125 disk.sda3.out‚ 176.80078125³©~™ˆ‚(:’ °ñçÔÖ¿¦y gpu.0.gpu‚85.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.215' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.642999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í©jÒ2‰- _wandbruntime‚5787  _runtime‚5787ÉÍ{BÇŠÁ & lr-AdamW/pg1‚0.0005969334579538986 & lr-AdamW/pg2‚0.0005969334579538986 trainer/global_step‚3249 % _timestamp‚1.7884755809390087e+09 _runtime‚5787.695238886 _step‚2622³ Ì‹Æ & lr-AdamW/pg1‚0.0005969334579538986 & lr-AdamW/pg2‚0.0005969334579538986 trainer/global_step‚3249 % _timestamp‚1.7884755809390087e+09 _runtime‚5787.695238886 _step‚262†q0Á 2Œ- _wandbruntime‚5787  _runtime‚5787ì²VAüO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚750 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚509110140419927 % _timestamp‚1.7884755809413025e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68349255 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501221806202672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81154072 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚513386902150144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚513386928541696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521289101653504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7888359645223 _step‚263 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68349261 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 _runtime‚5787.697615087 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7902199503360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72687775 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81180411 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7905810163712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501221806202672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81154072 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚513386902150144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚513386928541696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521289101653504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7888359645223 _step‚263 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68349261 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 _runtime‚5787.697615087 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7902199503360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72687775 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81180411 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7905810163712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132864 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72709412 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7908220658688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149548318 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149548318 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43185782 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 _runtime‚5789.741065079 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 trainer/global_step‚3249 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81178639 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464876557728256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501372135443808 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚513540706100736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149574689 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68369679 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464876506525184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509262875964349 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81205004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29524339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7894374611397 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149574689 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149548318 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43185782 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 _runtime‚5789.741065079 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 trainer/global_step‚3249 NADeviceStatsMonitor.oåüM7:n_train_batch_end/allocation.small_pool.freed‚81178639 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464876557728256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501372135443808 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚513540706100736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149574689 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68369679 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚464876506525184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509262875964349 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81205004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29524339 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7894374611397 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149574689 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72709412 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7908220658688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149548318 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529098518102528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29957240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚471701786788864 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚482752993533952 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82385774 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43813222 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚521075806493696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73771175 _step‚267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8026348660736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151782574 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 trainer/global_step‚3299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151756229 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚708 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚529098518102528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚521075806493696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚529102155154432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82385774 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69370455 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11051134806528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ô K=û ¨€P MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚521075806493696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73771175 _step‚267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8026348660736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151782574 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 trainer/global_step‚3299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151756229 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚708 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚529098518102528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚521075806493696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚529102155154432 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82385774 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69370455 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11051134806528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBA¬ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚529102155154432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚521075780102144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚482752870392320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚516750565322883 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8008688400731 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529098518102528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29957240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚471701786788864 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚482752993533952 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82385774 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43813222‹ûCÂý2©- _wandbruntime‚5886  _runtime‚5886Ü 'ª†N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11054526279168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 trainer/global_step‚3299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚529258343208448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69390885 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43826032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151827591 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151801220 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8028759155712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚521229557661184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73792832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82436706 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82410341 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚508888572072640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11054454353920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _step‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8028759155712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚516903300867513 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69390879 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73792133 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚471841077177856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151801220 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151827591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚482895480328704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29966106 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 % _timestamp‚1.7884756801671855e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69390885 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69390879 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚471841025974784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚516899666776657 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚508888597500608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚482895603457024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8025148482048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29966800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8014703366905 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82436706 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8025148482048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚521229584052736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712Éì¡'«‹N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚529254706143232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82410341 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151827591 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚482895480328704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29966106 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 % _timestamp‚1.7884756801671855e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69390885 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69390879 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚471841025974784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚516899666776657 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚508888597500608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚482895603457024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8025148482048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29966800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8014703366905 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82436706 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8025148482048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚521229584052736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123128320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11054526279168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71925248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 trainer/global_step‚3299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚529258343208448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69390885 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43826032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151827591 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151801220 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8028759155712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚521229557661184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73792832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82436706 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82410341 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚508888572072640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11054454353920 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _step‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8028759155712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚516903300867513 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69390879 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73792133 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚471841077177856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151801220 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚528764658054144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524383722044543 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70391655 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74852616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚516254730316272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11217272317952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚528764684445696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335936 trainer/global_step‚3349 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8143276497408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70391661 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536907934551552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚516254704888304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8143276497408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154035476 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70391661 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8129017156239 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83643815 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8146887157760 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚528764684445696ö È(ÀP \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚528764658054144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524383722044543 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70391655 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74852616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚516254730316272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11217272317952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚528764684445696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335936 trainer/global_step‚3349 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8143276497408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70391661 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536907934551552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚516254704888304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8143276497408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154035476 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70391661 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8129017156239 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83643815 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8146887157760 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚528764684445696 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424´}œœÁ‚(:‘ ‚óçÔ—ë§y gpu.0.gpu‚26.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚276.577' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.315400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ãržm‚(:â ‚óçÔ¦ÌÁ| disk.sda3.in‚0 disk.drbd1260.out‚ 0.8359375 disk.sda2.in‚0 network.sent‚ 8.639383e+06% disk./.usageGB‚1159.8959655761719! disk.nvme0c0n1.in‚ 16132.71875( disk.nvme0c0n1.out‚139516.1494140625 disk.sda3.out‚182.8125& disk.nvme0n1.out‚139516.1494140625 disk.sda2.out‚0 cpu‚25.081560665138195 network.recv‚ 1.897172e+06 disk.dm-1.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 16132.71875$ memory_percent‚89.62125778198242' proc.memory.availableMB‚ 1700.453125* proc.memory.percent‚19.194459915161133( disk./.usagePercent‚77.8415914673549 disk.dm-1.out‚182.8125 disk.sda.out‚182.8125 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3144.8203125 proc.cpu.threads‚17 disk.dm-0.in‚ 16132.71875# disk.dm-0.out‚139516.1494140625‰¡Ì\2Ã- _wandbruntime‚5985  _runtime‚5985ªy"ĆN ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚528918435613184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70412085 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70412079 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8149297652736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚490026177840640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884757785652487e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11220505093632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚537067759657472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8149297652736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74874318 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478805545424384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚528918462004736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚524540091679821 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 trainer/global_step‚3349 _runtime‚5985.323216539 _step‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154054122 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83668408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8131423459525 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154080493 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚716 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚537064122592256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154080493 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11220581213184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8135032122413 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154054122 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8145686979072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478805596627456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚490026050518016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83642043 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70412079 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚516405059557408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚528918462004736 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44466282 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74875034 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322624 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424ç«ñž'Å‹N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8145686979072 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8131423459525 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154080493 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚716 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚537064122592256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154080493 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11220581213184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8135032122413 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154054122 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8145686979072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478805596627456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚490026050518016 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83642043 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70412079 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚516405059557408 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚528918462004736 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44466282 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74875034 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322624 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚528918435613184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70412085 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70412079 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8149297652736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚490026177840640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884757785652487e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11220505093632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚537067759657472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8149297652736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74874318 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478805545424384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚528918462004736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚524540091679821 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 trainer/global_step‚3349 _runtime‚5985.323216539 _step‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154054122 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83668408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11382991619072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156262033 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚544720988052480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30840266 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485630774484992 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 trainer/global_step‚3399 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚544717351000576 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497013817307136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚544717351000576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8267425654784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75934719 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497013689971200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523771166945104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚536453562397696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚536453536006144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84849178 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71412861 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532024146947499 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30840992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71412855 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71412861 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84875517 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚544720988052480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552eί–ê RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523771192373072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚731 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45093727 _step‚275 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8249345911747 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156288378 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11382991619072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156262033 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚544720988052480 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30840266 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485630774484992 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 trainer/global_step‚3399 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚544717351000576 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497013817307136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚544717351000576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8267425654784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75934719 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497013689971200 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523771166945104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚536453562397696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚536453536006144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚726 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84849178 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71412861 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚532024146947499 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30840992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71412855 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71412861 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84875517 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚544720988052480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523771192373072 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424“FÌ^ß2Û- _wandbruntime‚6082  _runtime‚6082ôƒü7 '܆N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8255360877921 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8251752215033 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚523921521614208 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71433285 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚536607339956736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30849068 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485770064873984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8269836149760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11386233808896 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497156426005504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84900110 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84900110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚485770116077056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11386309928448 % _timestamp‚1.7884758761667418e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8266225476096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84873745 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8266225476096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75956290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8269836149760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156307024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84873745 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75955595 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30849758 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 trainer/global_step‚3399 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156333395 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71433285 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚536607313565184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚6082.923646803 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156307024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448ê»®!Ý‹N YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8269836149760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156307024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84873745 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75955595 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚276 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚690 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30849758 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 trainer/global_step‚3399 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156333395 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71433285 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚536607313565184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚6082.923646803 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156307024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8255360877921 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8251752215033 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚523921521614208 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71433285 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚536607339956736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30849068 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485770064873984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8269836149760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11386233808896 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497156426005504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84900110 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84900110 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚485770116077056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11386309928448 % _timestamp‚1.7884758761667418e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8266225476096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84873745 FctsMonitor.on_train_batch_end/active.all.allocated‚ 156333395 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8266225476096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75956290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187”ÚUõ2Þ- _wandbruntime‚6082  _runtime‚6082Êb`CZßÔ 1loss/SRqqt4/detection_loss‚2.5992364058891932 /loss/FRt1/detection_loss‚1.8943036800033914 0loss/FRt2/assignment_loss‚3.0783534840661653 /loss/FRt4/assignment_loss‚4.456462366827603 2loss/SRqqt2/assignment_loss‚2.9162575257451913 0loss/FBt4/assignment_loss‚0.5644415834775338 _runtime‚6082.925026234 1loss/FBt3/assignment_loss‚0.48068149216770606 .loss/FRt4/detection_loss‚3.140590717052591 trainer/global_step‚3399 /loss/FRt2/detection_loss‚2.1716477598766892 1loss/FBt2/assignment_loss‚0.23572397118581875 1loss/SRqqt3/detection_loss‚2.1310467140542135 /loss/FRt3/assignment_loss‚4.568094991502308 0loss/FBt1/assignment_loss‚0.2831686211607626 1loss/SRqqt2/detection_loss‚2.4545429066607825 /loss/FBt2/detection_loss‚0.8000807739794255 epoch‚2 $ _timestamp‚1.788475876169157e+09 _step‚277 .loss/FBt3/detection_loss‚2.484253396590551 0loss/FRt1/assignment_loss‚2.1001130683081493 1loss/SRqqt1/assignment_loss‚1.543857869666514 2loss/SRqqt4/assignment_loss‚2.2805303956071534 2loss/SRqqt3/assignment_loss‚2.0580610868831477 /loss/FBt1/detection_loss‚2.7330749763382807 .loss/FBt4/detection_loss‚2.374677149149088 %loss/total_loss‚52.07299185668518 1loss/SRqqt1/detection_loss‚1.1790724619539505 /loss/FRt3/detection_loss‚3.5447184585389637‰Óä_àÙ 1loss/SRqqt1/detection_loss‚1.1790724619539505 /loss/FRt3/detection_loss‚3.5447184585389637 1loss/SRqqt4/detection_loss‚2.5992364058891932 /loss/FRt1/detection_loss‚1.8943036800033914 0loss/FRt2/assignment_loss‚3.0783534840661653 /loss/FRt4/assignment_loss‚4.456462366827603 2loss/SRqqt2/assignment_loss‚2.9162575257451913 0loss/FBt4/assignment_loss‚0.5644415834775338 _runtime‚6082.925026234 1loss/FBt3/assignment_loss‚0.48068149216770606 .loss/FRt4/detection_loss‚3.140590717052591 trainer/global_step‚3399 /loss/FRt2/detection_loss‚2.1716477598766892 1loss/FBt2/assignment_loss‚0.23572397118581875 1loss/SRqqt3/detection_loss‚2.1310467140542135 /loss/FRt3/assignment_loss‚4.568094991502308 0loss/FBt1/assignment_loss‚0.2831686211607626 1loss/SRqqt2/detection_loss‚2.4545429066607825 /loss/FBt2/detection_loss‚0.8000807739794255 epoch‚2 $ _timestamp‚1.788475876169157e+09 _step‚277 .loss/FBt3/detection_loss‚2.484253396590551 0loss/FRt1/assignment_loss‚2.1001130683081493 1loss/SRqqt1/assignment_loss‚1.543857869666514 2loss/SRqqt4/assignment_loss‚2.2805303956071534 2loss/SRqqt3/assignment_loss‚2.0580610868831477 /loss/FBt1/detection_loss‚2.7330749763382807 .loss/FBt4/detection_loss‚2.374677149149088 %loss/total_loss‚52.07299185668518•.’š}|á‚(:ñ ëóçÔÊÒx disk.sda2.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1700.109375* proc.memory.percent‚19.194459915161133% disk./.usageGB‚1157.7504692077637 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 185.78125 disk.drbd1260.out‚ 0.8515625 disk.sda1.out‚0 cpu‚25.084694745613845 proc.cpu.threads‚17 network.recv‚ 1.93025e+06$ disk.nvme0c0n1.in‚16414.77734375) disk.nvme0c0n1.out‚141547.74951171875$ disk.dm-0.out‚141547.74951171875 disk.drbd1260.in‚0" disk.nvme0n1.in‚16414.77734375) disk./.usagePercent‚77.69760540587967 disk.dm-0.in‚16414.77734375 disk.dm-1.out‚ 185.78125 disk.sda3.out‚ 185.78125$ memory_percent‚89.62335586547852" proc.memory.rssMB‚ 3144.8203125 network.sent‚ 8.862947e+06 disk.sda3.in‚0' disk.nvme0n1.out‚141547.74951171875 disk.sda2.in‚0Έó6œâ‚(:‘ ëóçÔ Ôy gpu.0.gpu‚22.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚212.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.532199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾âàËxã‚(:í úóçÔ©½“x disk.sda1.out‚0$ memory_percent‚89.61367607116699 proc.cpu.threads‚17 disk.sda3.in‚0 network.sent‚ 8.970523e+06 network.recv‚ 1.935573e+06) disk./.usagePercent‚77.69754908422215% disk./.usageGB‚1157.7496299743652 disk.dm-0.in‚16414.77734375 disk.dm-1.in‚0 disk.dm-1.out‚186.1875 disk.sda3.out‚186.1875 disk.drbd1260.in‚0' disk.nvme0n1.out‚141750.42333984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 3147.1875 cpu‚25.085099031359224 disk.sda.in‚0$ disk.dm-0.out‚141750.42333984375" disk.nvme0n1.in‚16414.77734375( proc.memory.availableMB‚ 1701.6953125* proc.memory.percent‚19.208908081054688$ disk.nvme0c0n1.in‚16414.77734375) disk.nvme0c0n1.out‚141750.42333984375 disk.sda.out‚186.1875 disk.drbd1260.out‚ 0.8515625LÒŠä‚(:„ úóçÔÒ„×x gpu.0.gpu‚46.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[»Û;‚å‚(:÷ ‰ôçÔ®§‰x) disk./.usagePercent‚77.69597412587149 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 186.55078125 disk.sda3.in‚0 disk.sda3.out‚ 186.55078125 network.sent‚ 8.976696e+06) disk.nvme0c0n1.out‚141977.36376953125 disk.drbd1260.in‚0" disk.nvme0n1.in‚16414.77734375 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚89.61372375488281$ disk.dm-0.out‚141977.36376953125 disk.drbd1260.out‚ 0.85546875# proc.memory.rssMB‚ 3148.01171875) proc.memory.percent‚19.21393871307373 proc.cpu.threads‚17$ disk./.usageGB‚1157.726161956787$ disk.nvme0c0n1.in‚16414.77734375 disk.dm-0.in‚16414.77734375 disk.dm-1.out‚ 186.55078125 disk.sda.in‚0' disk.nvme0n1.out‚141977.36376953125% proc.memory.availableMB‚ 1701.6875 cpu‚25.08553776374409 network.recv‚ 1.94002e+06†º>3æ‚(:… ‰ôçÔ¬ÝÇx gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tâ{ç‚(:ð ˜ôçÔíš‚y disk.sda.in‚0 disk.sda.out‚186.9375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.85546875 disk.dm-0.in‚16519.66015625 disk.dm-1.out‚186.9375 network.recv‚ 1.943108e+06 disk.drbd1260.in‚0' disk.nvme0n1.out‚142229.65966796875 disk.sda2.in‚0) disk.nvme0c0n1.out‚142229.65966796875$ disk.dm-0.out‚142229.65966796875* proc.memory.percent‚19.225192070007324 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚186.9375" disk.nvme0n1.in‚16519.66015625$ disk.nvme0c0n1.in‚16519.66015625" memory_percent‚89.609694480896# proc.memory.rssMB‚ 3149.85546875) disk./.usagePercent‚77.68428405383547 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1702.34765625 cpu‚25.08604602936173 network.sent‚ 8.981525e+06% disk./.usageGB‚1157.5519714355469烎è‚(:„ ˜ôçÔ†Ó¬y gpu.0.gpu‚34.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.77' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.754 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NT÷8é‚(:’ §ôçÔú¯ôw gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚139.899' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.979799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›¸jyê‚(:î §ôçÔ®¹¤y disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚16519.66015625 disk.dm-1.out‚ 187.33984375 disk.sda3.out‚ 187.33984375# disk.nvme0n1.out‚142456.6796875) proc.memory.availableMB‚ 1702.34765625 cpu‚25.08651749203038 network.sent‚ 8.987921e+06) disk./.usagePercent‚77.68656021682241% disk./.usageGB‚1157.5858879089355 disk.dm-0.in‚16519.66015625 disk.sda.out‚ 187.33984375 disk.sda3.in‚0" disk.nvme0n1.in‚16519.66015625 disk.sda2.out‚0 network.recv‚ 1.948025e+06% disk.nvme0c0n1.out‚142456.6796875 disk.dm-1.in‚0 disk.drbd1260.in‚0" memory_percent‚89.609694480896# proc.memory.rssMB‚ 3149.85546875* proc.memory.percent‚19.225192070007324 disk.dm-0.out‚142456.6796875 disk.sda.in‚0 disk.drbd1260.out‚0.859375 disk.sda2.in‚0 proc.cpu.threads‚17©¹ ›2ë- _wandbruntime‚6164  _runtime‚6164r? ÇìÁ & lr-AdamW/pg1‚0.0005860323270530447 & lr-AdamW/pg2‚0.0005860323270530447 trainer/global_step‚3449 % _timestamp‚1.7884759578707557e+09 _runtime‚6164.628248144 _step‚278ß” ÌíÆ & lr-AdamW/pg1‚0.0005860323270530447 & lr-AdamW/pg2‚0.0005860323270530447 trainer/global_step‚3449 % _timestamp‚1.7884759578707557e+09 _runtime‚6164.628248144 _step‚278–5 ¢D2î- _wandbruntime‚6164  _runtime‚6164*–å}(ïûO trainer/global_step‚3449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚693 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚552526767449600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531287654429872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158514935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚552526767449600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72434055 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77014506 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539657303669159 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚544142413958144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504144265875968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45733977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77015199 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539660937759807 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531287629001904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8384353491456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚544142413958144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8373283329935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31280534 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 _step‚279 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÐXÕ(ð€P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86107219 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86107219 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8387964151808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚693 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚552526767449600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531287654429872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158514935 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚552526767449600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72434055 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77014506 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539657303669159 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚544142413958144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504144265875968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45733977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77015199 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539660937759807 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531287629001904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8384353491456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚544142413958144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8373283329935 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31280534 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 _step‚279 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72454485 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86105447 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8375689633429 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8386763973120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚552686592555520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86131812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158586297 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77036066 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504286737436160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539813673304437 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158586297 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158559926 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504286862661632 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72454479 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539810039213581 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11552153112576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 % _timestamp‚1.7884759589191682e+09 _runtime‚6165.678666523 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚544296191517184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492734584323584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45746777 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86131812UÞŽ'õ‹N OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72454479 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539810039213581 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11552153112576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 % _timestamp‚1.7884759589191682e+09 _runtime‚6165.678666523 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚544296191517184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492734584323584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45746777 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86131812 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚552686592555520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8386763973120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚552682955490304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531437958243040 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31289289 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 trainer/global_step‚3449 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72454485 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86105447 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8375689633429 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8386763973120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚552686592555520 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86131812 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158586297 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77036066 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090856 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504286737436160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539813673304437 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158586297 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158559926 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504286862661632 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160794182 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚499559864587264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8504891988480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547297728572115 trainer/global_step‚3499 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11715079800320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚560339820950528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch“pÂo@_start/inactive_split.small_pool.current‚707 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚560336183898624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8508502648832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚499559813384192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚538804116486672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73455255 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551831291910144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538804091058704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87338921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚551831318301696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31722072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚560336183898624 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _step‚283 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78096299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46374227 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31721365 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87338921 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6235.887116705 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551831318301696 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8508502648832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8504891988480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46374222 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 % _timestamp‚1.7884760291307924e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73455261 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87312582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424å{“Ä(†P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78095587 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511274944387584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160767837 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚560336183898624 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _step‚283 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78096299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46374227 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31721365 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87338921 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6235.887116705 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551831318301696 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8508502648832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8504891988480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46374222 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 % _timestamp‚1.7884760291307924e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73455261 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87312582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160794182 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚499559864587264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8504891988480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚547297728572115 trainer/global_step‚3499 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11715079800320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚560339820950528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚560336183898624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8508502648832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚499559813384192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚538804116486672 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73455255 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551831291910144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538804091058704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87338921 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚551831318301696 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31722072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87363514 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511417414840832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160812828 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78117109 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551985095860736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8507302470144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551985095860736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚560492371939328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚560496009004544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73475679 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚560496009004544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11718311067648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87337149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8510913143808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚499699103773184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160839199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87337149 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 $ _timestamp‚1.788476030224239e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73475685 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0\Y|Y‰ŠN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87363514 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551985069469184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46387032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551985069469184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125225472 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8496018388937 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511417540066304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11718311067648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87337149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8510913143808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚499699103773184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160839199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87337149 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 $ _timestamp‚1.788476030224239e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73475685 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547450464116745 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 _step‚284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46387027 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3499 _runtime‚6236.980677521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160812828 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87363514 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511417414840832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160812828 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78117109 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚551985095860736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8507302470144 MCDeviceStatsMonitor.on_train_batch_end/ pÙ·allocation.small_pool.current‚26365 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551985095860736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚560492371939328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚560496009004544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73475679 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚560496009004544 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚546320553115504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚554934519384423 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88544284 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74476461 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88570623 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11881485575168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚518405939453440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163020739 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568149237399552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163047084 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚506524332833792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8629041145856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47014472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 _step‚287 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚559520169862144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8613940840951 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8625430485504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74476455 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88570623 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚559520196253696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74476461 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79177573 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568145600347648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚506524384036864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚554930885293775 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 trainer/global_step‚3549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121044480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32162391 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚6315.249097462 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8629041145856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚559520196253696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8610332178271 $ _timestamp‚1.788476108493442e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74476455 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559520169862144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568145600347648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32163096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚518405818408960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88544284 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11881555416576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚546320578543472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568149237399552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79176863XdD_hž€P PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74476461 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79177573 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568145600347648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚506524384036864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚554930885293775 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 trainer/global_step‚3549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121044480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32162391 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚6315.249097462 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8629041145856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚559520196253696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8610332178271 $ _timestamp‚1.788476108493442e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74476455 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚559520169862144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568145600347648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32163096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚518405818408960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88544284 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11881555416576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚546320578543472 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568149237399552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79176863 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚546320553115504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚554934519384423 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88544284 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74476461 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88570623 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11881485575168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/ad—¤©ž llocated_bytes.large_pool.peak‚ 25702182912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚518405939453440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637051904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163020739 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568149237399552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163047084 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚506524332833792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8629041145856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47014472 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 _step‚287 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚559520169862144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8613940840951 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8625430485504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74476455 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88570623 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚559520196253696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ŸÚà$¤2Ÿ- _wandbruntime‚6316  _runtime‚6316´¡, ' †N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚518548571173376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555083620838197 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88568851 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8616347144445 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74496879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163092101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 _runtime‚6316.533258719 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74496879 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8612738481557 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚568305425453568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546470882356640 % _timestamp‚1.7884761097770927e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88595216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163065730 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32171815 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88568851 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8631451640832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163092101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121031168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74496885 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32171140 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _step‚288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555087254929053 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47027282 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8631451640832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88595216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚506663674425856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚506663623222784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568301788388352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚559673973812736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11884826919424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79198417 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69828096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47027277 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32001 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546470882356640 % _timestamp‚1.7884761097770927e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88595216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163065730 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32171815 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88568851 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8631451640832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163092101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121031168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74496885 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚675 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32171140 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _step‚288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555087254929053 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47027282 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8631451640832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88595216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚506663674425856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚506663623222784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568301788388352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚559673973812736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11884826919424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79198417 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69828096 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47027277 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚518548571173376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555083620838197 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88568851 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8616347144445 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74496879 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 žvÙ¨2¢- _wandbruntime‚6316  _runtime‚6316kìåZ£Ô 1loss/FBt2/assignment_loss‚0.12265984513461994 1loss/SRqqt1/detection_loss‚0.8337914689088897 /loss/FRt3/assignment_loss‚5.333566480212742 2loss/SRqqt2/assignment_loss‚2.2155160397291183 2loss/SRqqt3/assignment_loss‚2.3298496082425117 epoch‚2 1loss/FBt3/assignment_loss‚0.39049192360531987 /loss/FRt3/detection_loss‚3.4689242906040616 .loss/FBt3/detection_loss‚2.503592943151792 _runtime‚ 6316.53431563 _step‚289 0loss/FRt1/assignment_loss‚1.8393572473871536 %loss/total_loss‚50.56963144049749 trainer/global_step‚3549 .loss/FBt4/detection_loss‚2.615584240240209 0loss/FRt4/assignment_loss‚3.7029831602766707 2loss/SRqqt1/assignment_loss‚1.0437564781086817 1loss/FBt1/assignment_loss‚0.40438815691595664 .loss/FRt1/detection_loss‚1.680706545325545 1loss/SRqqt2/detection_loss‚2.5227610617876053 /loss/FBt2/detection_loss‚0.9063495191703759 /loss/FRt2/assignment_loss‚2.959433086713155 /loss/FRt2/detection_loss‚2.2393582973215316 1loss/SRqqt3/detection_loss‚2.2643368523567915 /loss/FBt1/detection_loss‚3.4182769805192947 % _timestamp‚1.7884761097788136e+09 1loss/SRqqt4/assignment_loss‚2.265798900808607 1loss/FBt4/assignment_loss‚0.13855366411563938 .loss/FRt4/detection_loss‚2.859437083875811 0loss/SRqqt4/detection_loss‚2.510157565985407¨¤T|_¤Ù _runtime‚ 6316.53431563 _step‚289 0loss/FRt1/assignment_loss‚1.8393572473871536 %loss/total_loss‚50.56963144049749 trainer/global_step‚3549 .loss/FBt4/detection_loss‚2.615584240240209 0loss/FRt4/assignment_loss‚3.7029831602766707 2loss/SRqqt1/assignment_loss‚1.0437564781086817 1loss/FBt1/assignment_loss‚0.40438815691595664 .loss/FRt1/detection_loss‚1.680706545325545 1loss/SRqqt2/detection_loss‚2.5227610617876053 /loss/FBt2/detection_loss‚0.9063495191703759 /loss/FRt2/assignment_loss‚2.959433086713155 /loss/FRt2/detection_loss‚2.2393582973215316 1loss/SRqqt3/detection_loss‚2.2643368523567915 /loss/FBt1/detection_loss‚3.4182769805192947 % _timestamp‚1.7884761097788136e+09 1loss/SRqqt4/assignment_loss‚2.265798900808607 1loss/FBt4/assignment_loss‚0.13855366411563938 .loss/FRt4/detection_loss‚2.859437083875811 0loss/SRqqt4/detection_loss‚2.510157565985407 1loss/FBt2/assignment_loss‚0.12265984513461994 1loss/SRqqt1/detection_loss‚0.8337914689088897 /loss/FRt3/assignment_loss‚5.333566480212742 2loss/SRqqt2/assignment_loss‚2.2155160397291183 2loss/SRqqt3/assignment_loss‚2.3298496082425117 epoch‚2 1loss/FBt3/assignment_loss‚0.39049192360531987 /loss/FRt3/detection_loss‚3.4689242906040616 .loss/FBt3/detection_loss‚2.503592943151792¡ßpÚ¥‚(:’ ÛõçÔ±–Äw gpu.0.gpu‚56.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚235.114' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚47.022800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð$󤄦‚(:ù ÛõçÔì³Ãx disk.sda3.in‚0 disk.sda3.out‚ 193.06640625" disk.nvme0n1.in‚16930.70703125) proc.memory.availableMB‚ 1702.21484375 network.recv‚ 2.002658e+06 disk.dm-0.in‚16930.70703125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8828125# proc.memory.rssMB‚ 3150.48046875) proc.memory.percent‚19.22900676727295 cpu‚25.089391164877224 network.sent‚ 9.340508e+06) disk./.usagePercent‚77.70358983800007 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 193.06640625& disk.nvme0n1.out‚145353.6591796875$ memory_percent‚89.61050510406494$ disk./.usageGB‚1157.839641571045$ disk.nvme0c0n1.in‚16930.70703125( disk.nvme0c0n1.out‚145353.6591796875# disk.dm-0.out‚145353.6591796875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 193.06640625 disk.sda.in‚0¹+zô’§‚(:‡ êõçÔ¡–›w gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚252.272' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.4544 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Ñ.€¨‚(:õ êõçÔÞ‡šx disk.sda2.out‚0( disk.nvme0c0n1.out‚145675.1982421875) proc.memory.availableMB‚ 1701.94140625$ disk./.usageGB‚1158.152141571045 disk.dm-0.in‚16930.70703125# disk.dm-0.out‚145675.1982421875$ memory_percent‚89.61217403411865* proc.memory.percent‚19.231128692626953 cpu‚25.089806606346414 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 193.4921875 disk.drbd1260.out‚ 0.8828125 disk.sda1.out‚0! proc.memory.rssMB‚ 3150.828125 network.sent‚ 9.448106e+06) disk./.usagePercent‚77.72456197520503 disk.dm-1.in‚0 disk.sda.out‚ 193.4921875 disk.drbd1260.in‚0& disk.nvme0n1.out‚145675.1982421875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚16930.70703125 network.recv‚ 2.008484e+06 disk.dm-1.out‚ 193.4921875 disk.sda.in‚0" disk.nvme0n1.in‚16930.70703125 disk.sda2.in‚0V¯Æφ©‚(:û ùõçÔ¦ËÆy network.recv‚ 2.013116e+06) disk.nvme0c0n1.out‚146094.27001953125$ disk.dm-0.out‚146094.27001953125! proc.memory.rssMB‚ 3150.828125* proc.memory.percent‚19.231128692626953 proc.cpu.threads‚17 network.sent‚ 9.454245e+06) disk./.usagePercent‚77.74291464331965" disk.nvme0n1.in‚16930.70703125 disk.sda1.in‚0$ disk.nvme0c0n1.in‚16930.70703125 disk.dm-1.in‚0$ memory_percent‚89.61102962493896 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚16930.70703125 disk.dm-1.out‚ 193.82421875 disk.sda.out‚ 193.82421875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.88671875$ disk./.usageGB‚1158.425609588623' disk.nvme0n1.out‚146094.27001953125 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.out‚ 193.82421875) proc.memory.availableMB‚ 1702.12890625 cpu‚25.09024568950086OdÌóª‚(:’ ùõçÔØæÐz gpu.0.gpu‚58.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚206.197' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.239399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 h‘8†«‚(:û ˆöçÔÞŠ“z disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚89.61100578308105$ disk.nvme0c0n1.in‚16930.70703125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚146344.74658203125 disk.sda1.in‚0 cpu‚25.090649716643483 network.sent‚ 9.459233e+06 disk.sda3.out‚ 194.26171875 disk.drbd1260.out‚ 0.88671875" disk.nvme0n1.in‚16930.70703125( proc.memory.availableMB‚ 1702.1328125! proc.memory.rssMB‚ 3150.828125* proc.memory.percent‚19.231128692626953 proc.cpu.threads‚17) disk.nvme0c0n1.out‚146344.74658203125 disk.dm-0.in‚16930.70703125 disk.dm-1.out‚ 194.26171875 disk.sda.out‚ 194.26171875 disk.sda1.out‚0 network.recv‚ 2.01631e+06) disk./.usagePercent‚77.75976198713293% disk./.usageGB‚1158.6766471862793$ disk.dm-0.out‚146344.74658203125 disk.sda.in‚0UÉj̬‚(:… ˆöçÔô×±z gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šHÓÄœ­‚(:‘ —öçÔ¡–úx gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.791000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K¿©v~®‚(:ó —öçÔ‚óåy) disk./.usagePercent‚77.80858595201052% disk./.usageGB‚1159.4041595458984 disk.sda.in‚0" disk.nvme0n1.in‚17048.62109375' disk.nvme0c0n1.out‚146595.408203125" disk.dm-0.out‚146595.408203125( proc.memory.availableMB‚ 1702.1328125* proc.memory.percent‚19.231128692626953 network.sent‚ 9.465639e+06 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.890625 disk.sda2.in‚0$ disk.nvme0c0n1.in‚17048.62109375 proc.cpu.threads‚17 network.recv‚ 2.021043e+06% disk.nvme0n1.out‚146595.408203125 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚17048.62109375$ memory_percent‚89.61100578308105! proc.memory.rssMB‚ 3150.828125 disk.dm-1.out‚ 194.70703125 disk.sda.out‚ 194.70703125 disk.sda3.out‚ 194.70703125 disk.sda1.out‚0 cpu‚25.09109822768671¸%À$¯‚(:† ¦öçÔ»¼ëz gpu.0.gpu‚33.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚203.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.6518 gpu.0.smCloZFn(mck‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬X y°‚(:î ¦öçÔæÄö{ network.recv‚ 2.024319e+06 disk.drbd1260.in‚0& disk.nvme0n1.out‚146990.3642578125$ disk.nvme0c0n1.in‚17048.62109375( disk.nvme0c0n1.out‚146990.3642578125 disk.dm-0.in‚17048.62109375* proc.memory.percent‚19.231128692626953 cpu‚25.0914863683753) disk./.usagePercent‚77.80845052402489 disk.sda3.out‚ 195.078125 disk.sda1.out‚0# disk.dm-0.out‚146990.3642578125 disk.dm-1.in‚0 disk.dm-1.out‚ 195.078125$ disk./.usageGB‚1159.402141571045 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 195.078125! proc.memory.rssMB‚ 3150.828125 proc.cpu.threads‚17 network.sent‚ 9.470525e+06 disk.sda3.in‚0 disk.drbd1260.out‚0.890625" disk.nvme0n1.in‚17048.62109375 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚89.61100578308105( proc.memory.availableMB‚ 1702.1328125ÚéS2±- _wandbruntime‚6412  _runtime‚64126RcDzÁ & lr-AdamW/pg1‚0.0005772853695111734 & lr-AdamW/pg2‚0.0005772853695111734 trainer/global_step‚3599 % _timestamp‚1.7884762053249066e+09 _runtime‚6412.080633493 _step‚290Þ~sÌ³Æ % _timestamp‚1.7884762053249066e+09 _runtime‚6412.080633493 _step‚290 & lr-AdamW/pg1‚0.0005772853695111734 & lr-AdamW/pg2‚0.0005772853695111734 trainer/global_step‚3599¢ŽCBŒ2´- _wandbruntime‚6412  _runtime‚6412–¥Šÿ'µùO MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553837040600272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚562571310196731 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8745968982528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32606435 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75497655 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚562567676106083 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚553837015172304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚567209074205696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8749579642880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75497655 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚567209047814144 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89775986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47654727 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89802325 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75497661 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚567209074205696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80261885 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚525536876809216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 trainer/global_step‚3599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165273641 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513488852283392 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8745968982528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32607158 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 % _timestamp‚1.7884762053257008e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8730660933779 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8734269596459 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47654722 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89775986 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165299986 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚525536751570432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513488903486464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12047973322752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚575955016796672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80261157 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424Þäôb(¶þO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚562567676106083 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚553837015172304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509786624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚567209074205696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8749579642880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75497655 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚567209047814144 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505157 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89775986 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47654727 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89802325 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766900 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75497661 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚567209074205696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80261885 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚525536876809216 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238784 trainer/global_step‚3599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165273641 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513488852283392 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8745968982528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32607158 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 % _timestamp‚1.7884762053257008e+09 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8730660933779 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8734269596459 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662680 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47654722 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89775986 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165299986 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚525536751570432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513488903486464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12047973322752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚575955016796672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80261157 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035712 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637051904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26339 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553837040600272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚562571310196731 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32001 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8745968982528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32606435 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32001 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75497655 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0£*?»Ô2·- _wandbruntime‚6413  _runtime‚6413®2| '¸†N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75518085 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8733067237065 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚567362825373184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576114841902592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8748379464192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80282721 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚567362851764736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165318632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553987369841408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165318632 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89826918 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚567362825373184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525679487084032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚525679361858560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165345003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513628142672384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553987344413440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89826918 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 trainer/global_step‚3599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75518079 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751990137856 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8748379464192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884762063400424e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89800553 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47667527 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12051293208576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89800553 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75518079 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751990137856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47667532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚576111204837376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚567362851764736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8736675899953 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75518085 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12051219186176ì 4•¹‹N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165318632 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89826918 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74022400 D;DeviceStatsMonitor.on_train_batch_e >b|$nd/segment.all.allocated‚3719 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚567362825373184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525679487084032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚525679361858560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165345003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513628142672384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553987344413440 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89826918 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610673664 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610673664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 trainer/global_step‚3599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75518079 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8751990137856 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8748379464192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884762063400424e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89800553 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47667527 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12051293208576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766900 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89800553 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75518079 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751990137856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47667532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚576111204837376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚567362851764736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8736675899953 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26365 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75518085 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12051219186176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75518085 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8733067237065 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26365 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚567362825373184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576114841902592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8748379464192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80282721 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚567362851764736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165318632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553987369841408 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662888 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509786624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505157¤¨Ë÷ª2º- _wandbruntime‚6413  _runtime‚6413´à‡@U»Ï 0loss/FBt2/assignment_loss‚0.1845025428171743 .loss/FRt1/detection_loss‚1.882447362247364 %loss/total_loss‚48.23155487256058 trainer/global_step‚3599 .loss/FRt3/assignment_loss‚4.57589898109436 2loss/SRqqt1/assignment_loss‚1.1094210080918856 /loss/FRt3/detection_loss‚3.4508266925811766 1loss/SRqqt2/detection_loss‚2.4067453157901766 $ _timestamp‚1.788476206343079e+09 /loss/FRt2/assignment_loss‚2.770605575393986 /loss/FBt1/detection_loss‚1.9627928206553826 _runtime‚6413.098673196 /loss/FRt4/assignment_loss‚3.958630260668303 0loss/FBt3/assignment_loss‚0.6388698353548534 2loss/SRqqt2/assignment_loss‚1.1635254102945327 2loss/SRqqt4/assignment_loss‚2.0391348460689187 1loss/FBt1/assignment_loss‚0.12872117866000876 /loss/FRt2/detection_loss‚2.6126411768632964 /loss/FRt4/detection_loss‚2.7431858432920357 1loss/SRqqt4/detection_loss‚2.1876818742603064 ,loss/FBt3/detection_loss‚2.4991151034832 0loss/FBt4/assignment_loss‚0.2609582438599318 1loss/SRqqt3/detection_loss‚2.6233469104766844 /loss/FBt2/detection_loss‚0.9303434439066608 0loss/FRt1/assignment_loss‚2.1192471170425415 epoch‚2 2loss/SRqqt3/assignment_loss‚1.9408470606803894 0loss/SRqqt1/detection_loss‚1.083506929280702 .loss/FBt4/detection_loss‚2.958559339696711 _step‚293äÝ`2Z¼Ô 2loss/SRqqt2/assignment_loss‚1.1635254102945327 2loss/SRqqt4/assignment_loss‚2.0391348460689187 1loss/FBt1/assignment_loss‚0.12872117866000876 /loss/FRt2/detection_loss‚2.6126411768632964 /loss/FRt4/detection_loss‚2.7431858432920357 1loss/SRqqt4/detection_loss‚2.1876818742603064 ,loss/FBt3/detection_loss‚2.4991151034832 0loss/FBt4/assignment_loss‚0.2609582438599318 1loss/SRqqt3/detection_loss‚2.6233469104766844 /loss/FBt2/detection_loss‚0.9303434439066608 0loss/FRt1/assignment_loss‚2.1192471170425415 epoch‚2 2loss/SRqqt3/assignment_loss‚1.9408470606803894 0loss/SRqqt1/detection_loss‚1.083506929280702 .loss/FBt4/detection_loss‚2.958559339696711 _step‚293 0loss/FBt2/assignment_loss‚0.1845025428171743 .loss/FRt1/detection_loss‚1.882447362247364 %loss/total_loss‚48.23155487256058 trainer/global_step‚3599 .loss/FRt3/assignment_loss‚4.57589898109436 2loss/SRqqt1/assignment_loss‚1.1094210080918856 /loss/FRt3/detection_loss‚3.4508266925811766 1loss/SRqqt2/detection_loss‚2.4067453157901766 $ _timestamp‚1.788476206343079e+09 /loss/FRt2/assignment_loss‚2.770605575393986 /loss/FBt1/detection_loss‚1.9627928206553826 _runtime‚6413.098673196 /loss/FRt4/assignment_loss‚3.958630260668303 0loss/FBt3/assignment_loss‚0.6388698353548534¥j`„ƒ½‚(:ø µöçÔ¾ê€z) disk.nvme0c0n1.out‚147254.59619140625 disk.dm-0.in‚17048.62109375 disk.dm-1.in‚0 disk.dm-1.out‚ 195.3671875 disk.sda.out‚ 195.3671875 disk.sda3.in‚0 disk.sda3.out‚ 195.3671875 disk.drbd1260.out‚ 0.8984375 network.sent‚ 9.476655e+06 network.recv‚ 2.028965e+06 disk.drbd1260.in‚0' disk.nvme0n1.out‚147254.59619140625 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚89.61148262023926( proc.memory.availableMB‚ 1702.0546875$ disk.nvme0c0n1.in‚17048.62109375$ disk.dm-0.out‚147254.59619140625 disk.sda.in‚0! proc.memory.rssMB‚ 3150.828125 cpu‚25.091904880938205) disk./.usagePercent‚77.80845103603995% disk./.usageGB‚1159.4021492004395" disk.nvme0n1.in‚17048.62109375 disk.sda2.in‚0 disk.sda1.in‚0* proc.memory.percent‚19.231128692626953 proc.cpu.threads‚17œPÔr¾‚(:„ µöçÔÒÛìz gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.86' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ `Å’¿‚(:‡ ÄöçÔõÈÊw gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.463' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.8926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ábÑBÀ‚(:ô ÄöçÔ¦Þx network.recv‚ 2.034843e+06) disk.nvme0c0n1.out‚147865.58837890625$ disk.dm-0.out‚147865.58837890625 disk.dm-1.out‚ 195.859375 disk.sda.in‚0 disk.sda3.out‚ 195.859375! proc.memory.rssMB‚ 3150.828125 cpu‚25.092435791962142% disk./.usageGB‚1159.4253120422363 disk.sda.out‚ 195.859375 disk.drbd1260.out‚ 0.8984375" disk.nvme0n1.in‚17048.62109375 disk.sda1.in‚0* proc.memory.percent‚19.231128692626953 proc.cpu.threads‚17 network.sent‚ 9.584123e+06 disk.drbd1260.in‚0' disk.nvme0n1.out‚147865.58837890625 disk.sda1.out‚0( proc.memory.availableMB‚ 1701.9609375) disk./.usagePercent‚77.81000551378787$ disk.nvme0c0n1.in‚17048.62109375 disk.dm-0.in‚17048.62109375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# memory_percent‚89.6120548248291¸ËÛ©Á‚(:’ ÓöçÔæë‹w gpu.0.gpu‚91.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚120.341' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.068199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÎU|‚(:ñ ÓöçÔüç•x disk.nvme0n1.in‚ 17172.46875! disk.nvme0c0n1.in‚ 17172.46875) disk.nvme0c0n1.out‚148188.65771484375) proc.memory.availableMB‚ 1701.45703125) disk./.usagePercent‚77.87725946105539 disk.sda3.out‚ 196.1640625 disk.drbd1260.in‚0 disk.sda2.in‚0* proc.memory.percent‚19.231128692626953 cpu‚25.092911180193013 proc.cpu.threads‚17 network.sent‚ 9.590519e+06% disk./.usageGB‚1160.4274444580078 disk.dm-0.in‚ 17172.46875$ disk.dm-0.out‚148188.65771484375 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 2.039828e+06 disk.sda3.in‚0' disk.nvme0n1.out‚148188.65771484375$ memory_percent‚89.61513042449951! proc.memory.rssMB‚ 3150.828125 disk.dm-1.out‚ 196.1640625 disk.sda.in‚0 disk.sda.out‚ 196.1640625 disk.drbd1260.out‚ 0.90234375ÔÜ óÂ(:’ âöçÔÈØ€x gpu.0.gpu‚77.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚139.367' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.873399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õcc¾~Ä‚(:ó âöçÔö´Þx network.sent‚ 9.595401e+06! disk.nvme0c0n1.in‚ 17172.46875) disk.nvme0c0n1.out‚148341.94677734375$ disk.dm-0.out‚148341.94677734375 disk.sda.in‚0 disk.sda3.out‚ 196.61328125 disk.sda2.in‚0) proc.memory.availableMB‚ 1701.45703125 disk.dm-1.out‚ 196.61328125 disk.sda.out‚ 196.61328125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.90234375 disk.nvme0n1.in‚ 17172.46875 disk.sda1.in‚0$ memory_percent‚89.61513042449951 network.recv‚ 2.04292e+06) disk./.usagePercent‚77.87712403306979% disk./.usageGB‚1160.4254264831543 disk.sda2.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3151.1796875 disk.dm-0.in‚ 17172.46875 disk.dm-1.in‚0' disk.nvme0n1.out‚148341.94677734375* proc.memory.percent‚19.233274459838867 cpu‚25.09330373827735 proc.cpu.threads‚17,K {Å‚(:ð ñöçÔÉ‘¢y' disk.nvme0n1.out‚148594.04052734375 disk.sda1.out‚0 disk.drbd1260.out‚0.90625# memory_percent‚90.2132511138916 cpu‚25.099891920724577 disk.nvme0n1.in‚ 17172.46875 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 17172.46875 disk.dm-1.in‚0 disk.dm-1.out‚ 196.87109375 disk.sda3.in‚0( proc.memory.availableMB‚ 1603.4609375" proc.memory.rssMB‚ 3249.1640625 network.sent‚ 9.601532e+06 network.recv‚ 2.047498e+06) disk./.usagePercent‚77.87554523460612% disk./.usageGB‚1160.4019012451172 disk.sda2.out‚0) disk.nvme0c0n1.out‚148594.04052734375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 17172.46875$ disk.dm-0.out‚148594.04052734375 disk.sda.out‚ 196.87109375 disk.sda3.out‚ 196.87109375* proc.memory.percent‚19.831323623657227 proc.cpu.threads‚171< oÆ‚(:… ñöçÔÚ„½y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4944 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëùýêmÇ‚(:â €÷çÔõö¯}) disk./.usagePercent‚77.84798397548488 disk.nvme0n1.in‚ 17172.46875) disk.nvme0c0n1.out‚149035.72021484375 disk.sda.in‚0 disk.sda3.out‚197.4375 disk.drbd1260.out‚0.90625# memory_percent‚90.2529239654541 network.recv‚ 2.050896e+06$ disk.dm-0.out‚149035.72021484375 disk.sda3.in‚0 cpu‚25.10585278914323 proc.cpu.threads‚17% disk./.usageGB‚1159.9912185668945' disk.nvme0n1.out‚149035.72021484375 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 17172.46875 disk.dm-1.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1596.9609375 disk.sda2.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 17172.46875 disk.dm-1.out‚197.4375 disk.sda.out‚197.4375! proc.memory.rssMB‚ 3255.296875* proc.memory.percent‚19.868755340576172 network.sent‚ 9.606623e+063½š‘È‚(:† €÷çÔÓÂ~ gpu.0.gpu‚34.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.004' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.6008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜Fœ7É‚(:… ÷çÔûÒÚw gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sgáj…Ê‚(:ú ÷çÔÍõ³z disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.percent‚19.88532543182373 cpu‚25.113734188636503 network.sent‚ 9.61491e+06% disk./.usageGB‚1159.9677505493164 disk.sda3.out‚ 198.03515625 disk.drbd1260.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 17299.8359375 disk.dm-0.in‚ 17299.8359375 disk.dm-1.out‚ 198.03515625 network.recv‚ 2.056076e+06 disk.sda.in‚0 disk.drbd1260.out‚ 0.91015625! disk.nvme0n1.in‚ 17299.8359375) disk.nvme0c0n1.out‚149478.73974609375$ disk.dm-0.out‚149478.73974609375$ memory_percent‚90.26868343353271) proc.memory.availableMB‚ 1594.37890625) disk./.usagePercent‚77.84640901713423 disk.sda.out‚ 198.03515625' disk.nvme0n1.out‚149478.73974609375 disk.dm-1.in‚0# proc.memory.rssMB‚ 3258.01171875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda2.in‚0Åê o‘Ë‚(:† ž÷çÔ©—ãw gpu.0.gpu‚31.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚278.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÌ‚(:ö ž÷çÔé•Ûy$ memory_percent‚90.28091430664062" proc.memory.rssMB‚ 3260.1796875 network.sent‚ 9.617874e+06 disk.dm-0.in‚ 17299.8359375 disk.sda3.out‚ 198.50390625$ proc.memory.availableMB‚1592.375* proc.memory.percent‚19.898557662963867 cpu‚25.121109206899376 proc.cpu.threads‚17 network.recv‚ 2.059725e+06 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 17299.8359375 disk.sda1.out‚0 disk.dm-1.out‚ 198.50390625% disk./.usageGB‚1159.9912490844727 disk.drbd1260.out‚ 0.91015625' disk.nvme0n1.out‚149897.65380859375 disk.sda2.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚149897.65380859375 disk.sda.in‚0 disk.sda.out‚ 198.50390625) disk./.usagePercent‚77.84798602354516 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 17299.8359375$ disk.dm-0.out‚149897.65380859375 disk.dm-1.in‚0 disk.sda3.in‚0o_TšÍ‚(:„ ­÷çÔ•Êy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚111.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.253 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m}ˆ‚΂(:÷ ­÷çÔ÷½»y disk.sda.in‚0 disk.sda3.out‚ 199.02734375) proc.memory.percent‚19.90382671356201' disk.nvme0n1.out‚150099.18798828125 disk.sda2.in‚0) disk.nvme0c0n1.out‚150099.18798828125 disk.dm-0.in‚ 17299.8359375 disk.sda3.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 1591.3671875# proc.memory.rssMB‚ 3261.04296875 disk.sda1.out‚0 disk.dm-1.out‚ 199.02734375 disk.sda.out‚ 199.02734375 disk.drbd1260.out‚ 0.9140625$ memory_percent‚90.28706550598145 cpu‚25.12942689677511 network.sent‚ 9.624109e+06 network.recv‚ 2.06426e+06! disk.nvme0n1.in‚ 17299.8359375 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 17299.8359375 proc.cpu.threads‚17) disk./.usagePercent‚77.84798602354516% disk./.usageGB‚1159.9912490844727 disk.sda1.in‚0$ disk.dm-0.out‚150099.18798828125 disk.dm-1.in‚0ò>Ž¿›Ï‚(: ¼÷çÔ¢¹¥w gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚276.539' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.30779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±£J@ƒÐ‚(:ø ¼÷çÔŸªåw disk.sda.in‚0 disk.sda3.in‚0 network.sent‚ 9.628884e+06 network.recv‚ 2.067417e+06# disk.nvme0c0n1.in‚ 17299.8359375$ disk.dm-0.out‚150349.65673828125 disk.dm-1.in‚0 disk.dm-1.out‚ 199.41796875) disk.nvme0c0n1.out‚150349.65673828125 disk.sda.out‚ 199.41796875 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 17299.8359375 disk.sda2.in‚0 disk.sda1.out‚0* proc.memory.percent‚19.907808303833008% disk./.usageGB‚1159.9677619934082 disk.dm-0.in‚ 17299.8359375 disk.drbd1260.out‚ 0.9140625' disk.nvme0n1.out‚150349.65673828125 disk.sda2.out‚0" proc.memory.rssMB‚ 3261.6953125) disk./.usagePercent‚77.84640978515684 disk.sda3.out‚ 199.41796875 disk.sda1.in‚0$ memory_percent‚90.27948379516602' proc.memory.availableMB‚ 1592.609375 cpu‚25.137191873209105 proc.cpu.threads‚179‡ë2Ñ- _wandbruntime‚6564  _runtime‚6564ü‚fÒãT OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚210 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 33395048448 _step‚294 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚576789010539520 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1532 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚48618025 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚48618022 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 167750417 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚76831968 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚534222159809024 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚562947031666800 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2706586276 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4507766900 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2090 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3921 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚585633106090496 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8844071256576 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚3719 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚585633106090496 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚90918444 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3896742400 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880505157 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 25168546816 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167750417 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 167729705 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 182188032 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚534221977620992 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4509786624 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚207 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2732853248 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32248 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚522053320852480 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3719 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1532 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚877 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚33151791 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 141470720 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2187 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32001 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2708558848 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚76831973 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708558848 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1221 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚571775619947555 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2187 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚576789010539520 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8828564949939 $ _timestamp‚1.788476357359054e+09 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚585630373237248 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20707 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚33151998 _runtime‚6564.116718655 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167729705 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8844071256576 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 8825858363663 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 4586471424 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚576789034833920 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20707 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_byta[d es.large_pool.current‚ 28808577024 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚562947054997616 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚522053361569792 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12168656768512 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880505157 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 25168546816 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167750417 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 167729705 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 182188032 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚534221977620992 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 4509786624 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚207 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2732853248 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32248 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚522053320852480 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3719 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1532 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚877 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚33151791 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 141470720 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2187 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32001 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2708558848 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚76831973 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708558848 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 4509786624 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1221 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚571775619947555 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2187 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚576789010539520 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 8828564949939 $ _timestamp‚1.788476357359054e+09 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚585630373237248 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20707 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚33151998 _runtime‚6564.116718655 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952¦*·ì2Ô- _wandbruntime‚6566  _runtime‚6566á¨!Os)ÕíR OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32001 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚12169348817408 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚2187 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4509786624 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2732853248 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚585738560325120 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2729917092 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 4509786624 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12169207346688 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 25702182912 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 167753995 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚2090 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 141470720 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚90906467 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20712 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 33395048448 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8844621834752 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚5 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚877 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 25168546816 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚2187 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32001 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8844621834752 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚81788171 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3719 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚571873455206051 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4586471424 trainer/global_step‚3626 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚90927174 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚76847533 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚208 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3896742400 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 $ _timestamp‚1.788476359391926e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32248 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚76847528 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507766900 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚585738560325120 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚81788382 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1532 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚33157807 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚76847528 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚40717312 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3719 _step‚295 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚585741293178368 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3921 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 182188032 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8826407301587 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚571876185123143 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚585741293178368 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2732853248 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8841913275904 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8841913275904 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚48630367 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 28880505157 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167753995 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3719 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 33395048448 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 $ _timestamp‚1.788476359391926e+09 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32248 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚76847528 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507766900 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚585738560325120 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚81788382 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1532 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚33157807 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚76847528 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚40717312 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3719 _step‚295 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚585741293178368 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3921 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 182188032 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 8826407301587 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚571876185123143 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚585741293178368 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2732853248 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8841913275904 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8841913275904 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚48630367 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 28880505157 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167753995 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3719 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 33395048448 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3719 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚571873455206051 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4586471424 trainer/global_step‚3626§Œ‰­„ׂ(:ù Ë÷çÔþÓÝx' disk.nvme0n1.out‚150744.64306640625 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚90.27984142303467* proc.memory.percent‚19.907808303833008) disk./.usagePercent‚77.84801904851707 disk.sda.out‚ 200.0390625 cpu‚25.145314156338877 proc.cpu.threads‚17) disk.nvme0c0n1.out‚150744.64306640625$ disk.dm-0.out‚150744.64306640625 disk.drbd1260.in‚0" disk.nvme0n1.in‚17412.76171875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1592.55078125" proc.memory.rssMB‚ 3261.6953125 disk.sda3.in‚0 disk.sda3.out‚ 200.0390625 network.sent‚ 9.635278e+06# disk./.usageGB‚1159.99174118042 disk.dm-0.in‚17412.76171875 disk.dm-1.in‚0 disk.dm-1.out‚ 200.0390625 disk.sda.in‚0 network.recv‚ 2.072346e+06$ disk.nvme0c0n1.in‚17412.76171875 disk.drbd1260.out‚ 0.91796875·sšØ‚(: Ë÷çÔÏ­ûy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚057]Ù‚(:„ Ú÷çÔÊ»øw gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.141 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tê‘Æ„Ú‚(:ù Ú÷çÔ×å¤{ disk.dm-0.in‚17412.76171875$ memory_percent‚90.28136730194092) proc.memory.availableMB‚ 1592.30078125 proc.cpu.threads‚17 disk.sda3.in‚0" disk.nvme0n1.in‚17412.76171875 disk.sda2.out‚0$ disk.nvme0c0n1.in‚17412.76171875) disk.nvme0c0n1.out‚151210.36572265625 disk.dm-1.out‚ 200.3046875 network.recv‚ 2.077773e+06% disk./.usageGB‚1159.9799919128418 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.dm-0.out‚151210.36572265625" proc.memory.rssMB‚ 3261.6953125 network.sent‚ 9.74451e+06) disk./.usagePercent‚77.84723054531162 disk.sda.out‚ 200.3046875 disk.sda3.out‚ 200.3046875 disk.drbd1260.out‚ 0.91796875 disk.dm-1.in‚0* proc.memory.percent‚19.907808303833008 cpu‚25.15226641929589 disk.sda.in‚0' disk.nvme0n1.out‚151210.36572265625»Ì2Û- _wandbruntime‚6591  _runtime‚6591ÅHZjÜפ _step‚296 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0904509648680687 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.44448569416999817 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5645893216133118 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_7‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.36771824955940247 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.20427927374839783 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19625122845172882 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0021028355695307255 ,jet/accuracy_3_of_3‚0.026434313505887985 particle/accuracy_4_of_4‚1 +particle/specificity‚0.9533593058586121 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999052882194519 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4573602080345154 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 particle/accuracy_6_of_7‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10006158798933029 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_5‚0.9188748598098755 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16218112409114838 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3228955566883087 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 +jet/accuracy_2_of_6‚0.23076923191547394 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24013745784759521 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9912084341049194 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.018460657447576523 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 trainer/global_step‚3626 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46420541405677795 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6145833134651184 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974597096443176 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1363869607448578 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13987432420253754 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10968703776597977 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5432412028312683 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47208040952682495 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.28727835416793823 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4034663140773773 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904374480247498 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1929599642753601 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.453091025352478 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.251851886510849 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.38425925374031067 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6811205744743347 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0381944440305233 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18005463480949402 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 particle/accuracy_3_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 % _timestamp‚1.7884763844290164e+09 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3147321045398712 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 jet/accuracy_6_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4948737621307373 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6007784008979797 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29160183668136597 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26273879408836365 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.029185868799686432 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15227697789669037 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9922122359275818 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4399999976158142 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3791535198688507 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4768062233924866 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.32701146602630615 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49675649404525757 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1410139948129654 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15028735995292664 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.42592594027519226 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.037380632013082504 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12090716511011124 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4147392213344574 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_2_of_4‚0.32637351751327515 particle/accuracy_1_of_4‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0070294784381985664 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15435878932476044 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_7‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989972710609436 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02425629086792469 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.508166491985321 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_2_of_3‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.043103449046611786 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46420541405677795 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2315656542778015 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41447368264198303 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6385116577148438 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32967546582221985 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34236109256744385 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 particle/accuracy_1_of_6‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.157525435090065 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07132457196712494 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15430590510368347 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30076807737350464 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5263974666595459 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9950795769691467 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2906317710876465 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46753326058387756 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40019381046295166 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19989721477031708 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9935243129730225 7validation_average_jet_accuracy‚0.41611286997795105 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5300925970077515 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5342040061950684 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49281615018844604 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6001218557357788 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21494251489639282 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1441524475812912 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5472797751426697 +jet/accuracy_2_of_2‚0.09879172593355179 particle/accuracy_3_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3703303337097168 *jet/accuracy_1_of_2‚0.7185203433036804 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 epoch‚2 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.510413646697998 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13050316274166107 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9866712093353271 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4164007306098938 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3891473114490509 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4804721772670746 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006844645366072655 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1412080079317093 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33663445711135864 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3539021611213684 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_1‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12546467781066895 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019274377264082432 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9904162287712097 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17978055775165558 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_2_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5013290047645569 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8108205199241638 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.42424246668815613 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14247985184192657 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018186913803219795 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14176243543624878 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921537041664124 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4047161340713501 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24534675478935242 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.015711722895503044 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 'particle/f_score‚0.3730178773403168 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3750000298023224 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10222222656011581 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.504770040512085 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_5‚0.4816756546497345 particle/accuracy_4_of_6‚1 particle/accuracy_1_of_5‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.30068227648735046 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12892000377178192 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11860669404268265 jet/accuracy_5_of_6‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920239448547363 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38087403774261475 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4537384510040283 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16319887340068817 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4066091775894165 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.40795767307281494 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48290956020355225 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04449404403567314 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4691358506679535 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1281694918870926 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13871106505393982 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43450212478637695 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2436862587928772 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03979261592030525 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24567806720733643 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26138779520988464 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4223787188529968 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12174297124147415 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3158648610115051 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 particle/accuracy_2_of_2‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1755441427230835 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *jet/accuracy_1_of_1‚0.4659838080406189 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16776509582996368 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3257717490196228 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.911347508430481 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 jet/accuracy_5_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4376010298728943 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5639474391937256 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2835511565208435 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49032631516456604 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FB7küÅ(t/FBt_purity‚0.9642857313156128 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2047206163406372 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.017328614369034767 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 (loss/val_total_loss‚47.9983467980782 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.482789546251297 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5619047284126282 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12962962687015533 +jet/accuracy_2_of_3‚0.22426803410053253 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5664682984352112 *jet/accuracy_1_of_4‚0.8783925771713257 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 jet/accuracy_3_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9843765497207642 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15227697789669037 *jet/accuracy_1_of_3‚0.8413580060005188 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5066666603088379 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17978395521640778 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5629628300666809 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37337538599967957 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23638786375522614 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18414486944675446 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14024943113327026 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.458010733127594 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876413941383362 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9970899820327759 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0021547081414610147 jet/accuracy_1_of_6‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _runtime‚6591.189258328 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3352678418159485 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1914682686328888 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9906396865844727 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.42518186569213867 jet/accuracy_5_of_5‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.565515398979187 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_4‚0.05207677558064461 +jet/accuracy_3_of_6‚0.03846153989434242 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.99339759349823 (particle/accuracy‚0.8493406176567078 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1975836604833603 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.042201101779937744 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5103585124015808 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30733880400657654 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2152777761220932 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5401003360748291 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877390265464783 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3379629850387573 +jet/accuracy_4_of_6‚0.03846153989434242 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03099730610847473 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007822733372449875 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3506944477558136 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2152777761220932 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1241438016295433 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30575886368751526 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.542453944683075 +jet/accuracy_3_of_5‚0.06957245618104935 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3676142692565918 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25254374742507935 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1353801190853119 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3397897481918335 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4363362193107605 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9938176274299622 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16934196650981903 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981371164321899 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2215750813484192 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01752811297774315 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5762566924095154 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004938271827995777 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776.\÷LîWÝܤ 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.4000000059604645 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.015711722895503044 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 'particle/f_score‚0.3730178773403168 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3750000298023224 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9874281287193298 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10222222656011581 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.504770040512085 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_5‚0.4816756546497345 particle/accuracy_4_of_6‚1 particle/accuracy_1_of_5‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.30068227648735046 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12892000377178192 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11860669404268265 jet/accuracy_5_of_6‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920239448547363 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.38087403774261475 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4537384510040283 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16319887340068817 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4066091775894165 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.40795767307281494 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48290956020355225 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04449404403567314 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4691358506679535 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1281694918870926 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13871106505393982 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43450212478637695 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2436862587928772 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03979261592030525 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24567806720733643 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26138779520988464 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9912280440330505 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4223787188529968 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12174297124147415 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3158648610115051 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 particle/accuracy_2_of_2‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1755441427230835 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 *jet/accuracy_1_of_1‚0.4659838080406189 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16776509582996368 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3257717490196228 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.911347508430481 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 jet/accuracy_5_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4376010298728943 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5639474391937256 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.2835511565208435 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49032631516456604 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2047206163406372 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.017328614369034767 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 (loss/val_total_loss‚47.9983467980782 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.482789546251297 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5619047284126282 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12962962687015533 +jet/accuracy_2_of_3‚0.22426803410053253 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5664682984352112 *jet/accuracy_1_of_4‚0.8783925771713257 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 jet/accuracy_3_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9843765497207642 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15227697789669037 *jet/accuracy_1_of_3‚0.8413580060005188 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.4000000059604645 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5066666603088379 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17978395521640778 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5629628300666809 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37337538599967957 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23638786375522614 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18414486944675446 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14024943113327026 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.458010733127594 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9876413941383362 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9970899820327759 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0021547081414610147 jet/accuracy_1_of_6‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _runtime‚6591.189258328 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3352678418159485 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1914682686328888 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9906396865844727 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.42518186569213867 jet/accuracy_5_of_5‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.565515398979187 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_4‚0.05207677558064461 +jet/accuracy_3_of_6‚0.03846153989434242 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.5 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.99339759349823 (particle/accuracy‚0.8493406176567078 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1975836604833603 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.042201101779937744 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5103585124015808 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30733880400657654 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2152777761220932 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5401003360748291 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877390265464783 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.0625 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3379629850387573 +jet/accuracy_4_of_6‚0.03846153989434242 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03099730610847473 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007822733372449875 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3506944477558136 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2152777761220932 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1241438016295433 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30575886368751526 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.542453944683075 +jet/accuracy_3_of_5‚0.06957245618104935 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3676142692565918 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25254374742507935 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1353801190853119 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3397897481918335 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4363362193107605 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9938176274299622 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16934196650981903 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981371164321899 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2215750813484192 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.01752811297774315 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5762566924095154 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004938271827995777 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4999999701976776 _step‚296 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0904509648680687 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.44448569416999817 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5645893216133118 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_7‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.36771824955940247 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.20427927374839783 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19625122845172882 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0021028355695307255 ,jet/accuracy_3_of_3‚0.026434313505887985 particle/accuracy_4_of_4‚1 +particle/specificity‚0.9533593058586121 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999052882194519 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.979166567325592 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4573602080345154 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 particle/accuracy_6_of_7‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10006158798933029 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_1_of_5‚0.9188748598098755 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16218112409114838 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3228955566883087 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 +jet/accuracy_2_of_6‚0.23076923191547394 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24013745784759521 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9912084341049194 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.018460657447576523 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 trainer/global_step‚3626 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46420541405677795 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6145833134651184 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974597096443176 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1363869607448578 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13987432420253754 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.596491277217865 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10968703776597977 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5432412028312683 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47208040952682495 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1721850Aª6(u:18658638 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,particle/sensitivity‚0.28727835416793823 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4034663140773773 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904374480247498 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1929599642753601 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.453091025352478 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.251851886510849 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.38425925374031067 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6811205744743347 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0381944440305233 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18005463480949402 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.38333332538604736 particle/accuracy_3_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 % _timestamp‚1.7884763844290164e+09 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3147321045398712 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 jet/accuracy_6_of_7‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4948737621307373 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6007784008979797 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29160183668136597 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26273879408836365 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.029185868799686432 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15227697789669037 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9922122359275818 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4399999976158142 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3791535198688507 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4768062233924866 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.32701146602630615 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49675649404525757 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1410139948129654 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15028735995292664 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.42592594027519226 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.037380632013082504 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12090716511011124 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4147392213344574 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 +jet/accuracy_2_of_4‚0.32637351751327515 particle/accuracy_1_of_4‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0070294784381985664 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15435878932476044 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_7‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989972710609436 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02425629086792469 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.508166491985321 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_2_of_3‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.043103449046611786 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46420541405677795 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2315656542778015 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.30000001192092896 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41447368264198303 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6385116577148438 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.32967546582221985 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34236109256744385 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 particle/accuracy_1_of_6‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 6 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.157525435090065 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07132457196712494 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15430590510368347 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30076807737350464 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5263974666595459 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9950795769691467 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2906317710876465 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46753326058387756 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40019381046295166 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19989721477031708 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9935243129730225 7validation_average_jet_accuracy‚0.41611286997795105 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5300925970077515 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5342040061950684 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49281615018844604 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6001218557357788 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21494251489639282 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1441524475812912 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5472797751426697 +jet/accuracy_2_of_2‚0.09879172593355179 particle/accuracy_3_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3703303337097168 *jet/accuracy_1_of_2‚0.7185203433036804 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 epoch‚2 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.510413646697998 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13050316274166107 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9866712093353271 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4164007306098938 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3891473114490509 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4804721772670746 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006844645366072655 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1412080079317093 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33663445711135864 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3539021611213684 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_1‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12546467781066895 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019274377264082432 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9904162287712097 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_7_of_7‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17978055775165558 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 particle/accuracy_2_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5013290047645569 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8108205199241638 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.125 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.42424246668815613 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.125 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14247985184192657 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018186913803219795 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14176243543624878 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921537041664124 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4047161340713501 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24534675478935242¨_¹}Þ‚(:ò é÷çÔ†»êy disk.sda.out‚ 200.68359375 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.156713741900692 disk.dm-0.in‚17412.76171875 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚17412.76171875$ memory_percent‚99.43389892578125! proc.memory.availableMB‚92.75" proc.memory.rssMB‚ 4758.2265625 network.sent‚ 9.748717e+06 network.recv‚ 2.081953e+06) disk./.usagePercent‚77.84644229811369$ disk.dm-0.out‚151701.47705078125 disk.sda3.out‚ 200.68359375' disk.nvme0n1.out‚151701.47705078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.drbd1260.out‚0.921875$ disk.nvme0c0n1.in‚17412.76171875) disk.nvme0c0n1.out‚151701.47705078125) proc.memory.percent‚29.04191017150879 proc.cpu.threads‚17$ disk./.usageGB‚1159.968246459961 disk.dm-1.out‚ 200.68359375 ™>šß‚(: é÷çÔ¹†ž{ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚31.0 gpu.0.powerWatts‚91.923' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.384600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¾y~à‚(:ó ø÷çÔíõ¥y disk.dm-1.in‚0 disk.sda.out‚ 201.09375 disk.sda3.in‚0$ memory_percent‚85.43031215667725 network.recv‚ 2.087856e+06' disk.nvme0n1.out‚156062.52685546875 disk.sda2.in‚0 disk.dm-1.out‚ 201.09375 disk.sda.in‚0 disk.sda3.out‚ 201.09375# proc.memory.rssMB‚ 5858.83203125) proc.memory.percent‚35.75947284698486) disk./.usagePercent‚78.03374637042758 disk.sda1.in‚0$ disk.nvme0c0n1.in‚17412.76171875) disk.nvme0c0n1.out‚156062.52685546875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.921875) proc.memory.availableMB‚ 2387.09765625% disk./.usageGB‚1162.7592124938965 disk.dm-0.in‚17412.76171875$ disk.dm-0.out‚156062.52685546875 cpu‚25.155213079579916 proc.cpu.threads‚17 network.sent‚ 9.871682e+06" disk.nvme0n1.in‚17412.76171875 disk.sda2.out‚0 disk.sda1.out‚08©wvšá‚(: ø÷çÔ£y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B›É·â‚(:ö ‡øçÔ—€‘{ disk.sda.out‚ 201.55859375 disk.sda3.out‚ 201.55859375$ memory_percent‚68.26891899108887 network.recv‚ 2.091906e+06$ disk./.usageGB‚1163.994773864746 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚160526.7392578125 disk.dm-0.in‚ 17479.7578125 disk.sda.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 17479.7578125# disk.dm-0.out‚160526.7392578125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.92578125( proc.memory.availableMB‚ 5198.8203125" proc.memory.rssMB‚ 3263.8515625 cpu‚25.154890091234513 proc.cpu.threads‚17 network.sent‚ 9.877207e+06! disk.nvme0n1.in‚ 17479.7578125& disk.nvme0n1.out‚160526.7392578125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 201.55859375* proc.memory.percent‚19.920969009399414) disk./.usagePercent‚78.11666593073035 disk.sda2.in‚0Û«9™ã‚(:… ‡øçÔ…º‡| gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚34.0 gpu.0.powerWatts‚170.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0882 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'9–bœä‚(:‘ –øçÔ’Ô±{ gpu.0.gpu‚41.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.635' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.526999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^_!~å‚(:ó –øçÔƒ¶ñ| disk.dm-1.in‚0 disk.dm-1.out‚ 201.96484375 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 17532.828125 disk.sda1.in‚0 disk.dm-0.in‚ 17532.828125 disk.sda.out‚ 201.96484375* proc.memory.percent‚19.920969009399414 proc.cpu.threads‚17 network.sent‚ 9.882053e+06( disk./.usagePercent‚78.1150909723797 disk.sda3.out‚ 201.96484375 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 17532.828125( disk.nvme0c0n1.out‚160777.2119140625$ memory_percent‚68.27070713043213 disk.drbd1260.out‚ 0.92578125 disk.sda2.out‚0# disk.dm-0.out‚160777.2119140625) proc.memory.availableMB‚ 5198.52734375" proc.memory.rssMB‚ 3263.8515625 cpu‚25.154754330481587 network.recv‚ 2.095626e+06$ disk./.usageGB‚1163.971305847168 disk.drbd1260.in‚0& disk.nvme0n1.out‚160777.2119140625 disk.sda1.out‚0ÁXJ~æ‚(:ó ¥øçÔÑ£| disk.sda.out‚ 202.32421875$ memory_percent‚68.27287673950195" proc.memory.rssMB‚ 3263.8515625* proc.memory.percent‚19.920969009399414 network.recv‚ 2.100039e+06) disk.nvme0c0n1.out‚161122.92529296875 disk.sda3.in‚0 disk.drbd1260.out‚ 0.9296875 disk.nvme0n1.in‚ 17532.828125' disk.nvme0n1.out‚161122.92529296875 disk.sda1.in‚0' proc.memory.availableMB‚ 5198.171875" disk.nvme0c0n1.in‚ 17532.828125$ disk.dm-0.out‚161122.92529296875 disk.dm-1.in‚0 disk.dm-1.out‚ 202.32421875 disk.sda2.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.11508610823654 disk.dm-0.in‚ 17532.828125 disk.sda3.out‚ 202.32421875 disk.drbd1260.in‚0 disk.sda2.out‚0 cpu‚25.154748549785435 proc.cpu.threads‚17 network.sent‚ 9.888222e+06# disk./.usageGB‚1163.97123336792 disk.sda.in‚04Ú’ç‚(:‡ ¥øçÔ˜Ôš~ gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚196.283' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.2566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0מ2è- _wandbruntime‚6665  _runtime‚6665‰s3jÄé¾ % lr-AdamW/pg1‚0.000574264129927629 % lr-AdamW/pg2‚0.000574264129927629 trainer/global_step‚3649 $ _timestamp‚1.788476459074273e+09 _runtime‚6665.830675528 _step‚297(ž@Éêà % lr-AdamW/pg1‚0.000574264129927629 % lr-AdamW/pg2‚0.000574264129927629 trainer/global_step‚3649 $ _timestamp‚1.788476459074273e+09 _runtime‚6665.830675528 _step‚297©`vNÔ2ë- _wandbruntime‚6665  _runtime‚6665("ø(ìüO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67743744 % _timestamp‚1.7884764590759082e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77499141 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚567654525629904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚581679005649408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚581678979257856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82502992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590580190152192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169085094 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77499135 _step‚298 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33430411 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12251344952320 _runtime‚6665.831973896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91585953 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚567654500201936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8901210894336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8904821555200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169058748 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77499135è?’—·íP TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91585953 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train €[ÍP&_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚567654500201936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8901210894336 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8904821555200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169058748 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77499135 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚67743744 % _timestamp‚1.7884764590759082e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77499141 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚567654525629904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚581679005649408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚581678979257856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82502992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590580190152192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169085094 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77499135 _step‚298 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33430411 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12251344952320 _runtime‚6665.831973896ªŒóE=2î- _wandbruntime‚6666  _runtime‚6666GÁÞÉ 'ï†N XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8903621376000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12254708527104 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884764601315193e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚581832783208448 _step‚299 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚538844444191744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚576692825632230 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91610546 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12254640796672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 _runtime‚6666.887811226 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚581832756816896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169130111 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590736378192896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8907232050176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526589684461568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91584180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77519565 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77519559 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91584180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚581832783208448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82524651 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8903621376000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8891604852050 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77519559 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33439265 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚590740015258624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚590740015258624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77519565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49085386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49085381 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚576696459723090 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169103739 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82523973 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 36340908603òˆÀ'ð‹N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91610546 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12254640796672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 _runtime‚6666.887811226 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚581832756816896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169130111 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590736378192896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8907232050176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526589684461568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91584180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77519565 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77519559 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91584180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚581832783208448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82524651 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8903621376000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8891604852050 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77519559 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33439265 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚590740015258624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚590740015258624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77519565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49085386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49085381 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚576696459723090 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169103739 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82523973 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8903621376000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12254708527104 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884764601315193e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚581832783208448 _step‚299 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚538844444191744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚576692825632230«ƒw2ñ- _wandbruntime‚6666  _runtime‚6666‚Ì•`ZòÔ /loss/FRt4/detection_loss‚3.0767985454627444 1loss/SRqqt4/detection_loss‚2.6204915821552275 .loss/FBt3/detection_loss‚2.265923920131865 /loss/FBt1/detection_loss‚2.6352371248331936 trainer/global_step‚3649 /loss/FRt3/assignment_loss‚5.004846251529196 2loss/SRqqt2/assignment_loss‚1.6051900126039982 0loss/FBt2/assignment_loss‚0.2950020932657446 /loss/FBt4/detection_loss‚2.4723500188659218 % _timestamp‚1.7884764601335623e+09 0loss/FRt2/assignment_loss‚2.4313273623585703 1loss/FBt4/assignment_loss‚0.22575516670065307 /loss/FBt2/detection_loss‚0.7264346146423902 epoch‚3 0loss/SRqqt2/detection_loss‚2.300630788008372 _runtime‚ 6666.88918707 /loss/FRt2/detection_loss‚2.0860654106363654 0loss/SRqqt3/detection_loss‚2.938815028965473 _step‚300 /loss/FRt4/assignment_loss‚3.965957684176309 /loss/FRt1/detection_loss‚1.7191934835397964 1loss/SRqqt1/detection_loss‚1.1436891323990293 %loss/total_loss‚49.08816253565925 1loss/SRqqt1/assignment_loss‚1.224927489558219 2loss/SRqqt3/assignment_loss‚1.9048819363117218 1loss/FBt3/assignment_loss‚0.36949807260992074 .loss/FRt3/detection_loss‚3.209656217823858 0loss/FRt1/assignment_loss‚2.3736651559670765 2loss/SRqqt4/assignment_loss‚2.1667158395051955 1loss/FBt1/assignment_loss‚0.32510960360840807ĘÀ_óÙ 0loss/FRt1/assignment_loss‚2.3736651559670765 2loss/SRqqt4/assignment_loss‚2.1667158395051955 1loss/FBt1/assignment_loss‚0.32510960360840807 /loss/FRt4/detection_loss‚3.0767985454627444 1loss/SRqqt4/detection_loss‚2.6204915821552275 .loss/FBt3/detection_loss‚2.265923920131865 /loss/FBt1/detection_loss‚2.6352371248331936 trainer/global_step‚3649 /loss/FRt3/assignment_loss‚5.004846251529196 2loss/SRqqt2/assignment_loss‚1.6051900126039982 0loss/FBt2/assignment_loss‚0.2950020932657446 /loss/FBt4/detection_loss‚2.4723500188659218 % _timestamp‚1.7884764601335623e+09 0loss/FRt2/assignment_loss‚2.4313273623585703 1loss/FBt4/assignment_loss‚0.22575516670065307 /loss/FBt2/detection_loss‚0.7264346146423902 epoch‚3 0loss/SRqqt2/detection_loss‚2.300630788008372 _runtime‚ 6666.88918707 /loss/FRt2/detection_loss‚2.0860654106363654 0loss/SRqqt3/detection_loss‚2.938815028965473 _step‚300 /loss/FRt4/assignment_loss‚3.965957684176309 /loss/FRt1/detection_loss‚1.7191934835397964 1loss/SRqqt1/detection_loss‚1.1436891323990293 %loss/total_loss‚49.08816253565925 1loss/SRqqt1/assignment_loss‚1.224927489558219 2loss/SRqqt3/assignment_loss‚1.9048819363117218 1loss/FBt3/assignment_loss‚0.36949807260992074 .loss/FRt3/detection_loss‚3.209656217823858¬Wz_œ>ô‚(:‘ ´øçÔóëÎz gpu.0.gpu‚29.0 gpu.0.memory‚Vp.^4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚150.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.182199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%WC­tõ‚(:é ´øçÔµÖú{ disk.sda3.in‚0$ disk.dm-0.out‚161565.62451171875 disk.dm-1.in‚0 disk.sda.out‚202.6875 disk.drbd1260.out‚ 0.9296875 disk.nvme0n1.in‚ 17532.828125" proc.memory.rssMB‚ 3263.8515625* proc.memory.percent‚19.920969009399414 network.recv‚ 2.103399e+06" disk.nvme0c0n1.in‚ 17532.828125 disk.sda3.out‚202.6875 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚68.27800273895264 proc.cpu.threads‚17( disk./.usagePercent‚78.1150909723797$ disk./.usageGB‚1163.971305847168) disk.nvme0c0n1.out‚161565.62451171875 disk.dm-0.in‚ 17532.828125 disk.sda.in‚0' disk.nvme0n1.out‚161565.62451171875 cpu‚25.154924841908226 network.sent‚ 9.893172e+06 disk.dm-1.out‚202.6875 disk.sda2.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 5197.33203125ˆéWö‚(:ö ÃøçÔÅ™ðy) disk.nvme0c0n1.out‚162008.64404296875 disk.drbd1260.out‚ 0.93359375 disk.nvme0n1.in‚ 17532.828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 5197.54296875$ disk.dm-0.out‚162008.64404296875 disk.sda3.in‚0' disk.nvme0n1.out‚162008.64404296875$ memory_percent‚68.27671527862549" proc.memory.rssMB‚ 3263.8515625 proc.cpu.threads‚17 disk.dm-1.out‚ 203.2578125 disk.sda.in‚0 disk.sda.out‚ 203.2578125 disk.sda1.out‚0* proc.memory.percent‚19.920969009399414 cpu‚25.155187648255396 network.sent‚ 1.0002343e+07) disk./.usagePercent‚78.11666618673787% disk./.usageGB‚1163.9947776794434" disk.nvme0c0n1.in‚ 17532.828125 disk.dm-0.in‚ 17532.828125 disk.dm-1.in‚0 disk.sda3.out‚ 203.2578125 disk.drbd1260.in‚0 network.recv‚ 2.110575e+06"B6‡’÷‚(:‡ ÃøçÔû•‰z gpu.0.gpu‚55.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚213.948' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.7896 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­lúœø‚(:‘ ÒøçÔàçƒx gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚279.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ï#ßyù‚(:î ÒøçÔó‘y) proc.memory.availableMB‚ 5197.53515625 cpu‚25.15521855253929' disk.nvme0c0n1.out‚162354.431640625% disk.nvme0n1.out‚162354.431640625" proc.memory.rssMB‚ 3263.8515625 proc.cpu.threads‚17 network.recv‚ 2.113775e+06) disk./.usagePercent‚78.06813202238907! disk.nvme0c0n1.in‚ 17671.28125 disk.dm-1.out‚ 203.51171875 disk.sda.in‚0 disk.sda.out‚ 203.51171875 disk.drbd1260.in‚0* proc.memory.percent‚19.920969009399414 network.sent‚ 1.0007117e+07 disk.dm-0.in‚ 17671.28125" disk.dm-0.out‚162354.431640625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.93359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ disk./.usageGB‚1163.271583557129 disk.sda3.in‚0 disk.sda3.out‚ 203.51171875 disk.nvme0n1.in‚ 17671.28125 disk.sda1.out‚0$ memory_percent‚68.27676296234131£b¸‡zú‚(:ï áøçÔ…×ðw network.sent‚ 1.0013243e+07! disk.nvme0c0n1.in‚ 17671.28125 disk.dm-0.in‚ 17671.28125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 network.recv‚ 2.11823e+06$ disk.dm-0.out‚162668.90283203125 disk.dm-1.out‚ 203.85546875 disk.sda.out‚ 203.85546875 disk.drbd1260.in‚0' disk.nvme0n1.out‚162668.90283203125 disk.sda1.in‚0 disk.sda3.out‚ 203.85546875) disk./.usagePercent‚78.07452657857932% disk./.usageGB‚1163.3668670654297) disk.nvme0c0n1.out‚162668.90283203125 disk.sda2.out‚0 disk.sda1.out‚0# memory_percent‚68.2783842086792" proc.memory.rssMB‚ 3263.8515625 disk.sda.in‚0 disk.drbd1260.out‚0.9375 disk.nvme0n1.in‚ 17671.28125) proc.memory.availableMB‚ 5197.26953125* proc.memory.percent‚19.920969009399414 cpu‚25.15510674537775 proc.cpu.threads‚17.ÄÚ(û‚(:’ áøçÔ Ÿ¶x gpu.0.gpu‚33.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚273.818' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.763600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¬ÇÖœü‚(:‘ ðøçÔ±ç´w gpu.0.gpu‚64.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚273.676' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&0w±yý‚(:î ðøçÔÙºx disk.dm-0.in‚ 17671.28125 disk.sda.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚162909.7900390625 disk.sda.out‚ 204.26171875 disk.drbd1260.out‚0.9375$ memory_percent‚68.27685832977295" proc.memory.rssMB‚ 3263.8515625 cpu‚25.15494807021572 proc.cpu.threads‚17 disk.nvme0n1.in‚ 17671.28125# disk.dm-0.out‚162909.7900390625 disk.dm-1.out‚ 204.26171875 disk.sda3.in‚0 disk.drbd1260.in‚0) proc.memory.availableMB‚ 5197.51953125 network.sent‚ 1.0018128e+07) disk./.usagePercent‚78.07452939466219 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 204.26171875* proc.memory.percent‚19.920969009399414 network.recv‚ 2.121333e+06% disk./.usageGB‚1163.3669090270996& disk.nvme0n1.out‚162909.7900390625 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 17671.28125Näu2þ- _wandbruntime‚6735  _runtime‚6735áP3ÇÿÁ & lr-AdamW/pg2‚0.0005711911370603705 trainer/global_step‚3699 % _timestamp‚1.7884765290576339e+09 _runtime‚6735.813148867 _step‚301 & lr-AdamW/pg1‚0.0005711911370603705'n Ì€Æ _step‚301 & lr-AdamW/pg1‚0.0005711911370603705 & lr-AdamW/pg2‚0.0005711911370603705 trainer/global_step‚3699 % _timestamp‚1.7884765290576339e+09 _runtime‚6735.813148867­·_52- _wandbruntime‚6735  _runtime‚6735%.ú(‚ûO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78520335 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9025360052224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171337996 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545833146946560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6735.814308052 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33875592 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12418182221312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9005918641380 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚598393243653632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9021749391360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12418112380416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83588423 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92817655 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78520335 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚598389606601216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92791315 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33874929 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚584180514990768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92791315 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚545833025902592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚584176880900116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚533414913522176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575170987686704 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 _step‚302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9021749391360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589367857209856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92817655 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575170962258736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589367883601408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69840896 $ _timestamp‚1.788476529058664e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78520341 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9009527304064 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚533414964725248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 trainer/global_step‚3699ß´kŸ(ƒ€P PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78520335 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚598389606601216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92791315 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚668 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33874929 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚584180514990768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92791315 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚545833025902592 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚584176880900116 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚533414913522176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575170987686704 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 _step‚302 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9021749391360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589367857209856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92817655 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575170962258736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589367883601408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69840896 $ _timestamp‚1.788476529058664e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78520341 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9009527304064 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚533414964725248 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 trainer/global_step‚3699 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78520335 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9025360052224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171337996 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545833146946560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6735.814308052 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33875592 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12418182221312 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9005918641380 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚598393243653632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9021749391360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12418112380416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83588423 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92817655 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912®[˜TÜ2„- _wandbruntime‚6736  _runtime‚6736L¸H,……N MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚584333250535398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9024159873024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171356641 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49725636 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171383013 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589521634768896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589521634768896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171383013 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚584329616444538 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589521661160448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545975664502272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171356641 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92815882 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589521661160448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78540765 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12421530418688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚598549431707648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9011933607558 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚545975785532928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ;5DeviceSta@‘mßtsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3699 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12421460591104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575321316927840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9008324944666 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83609487 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚598545794641920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92815882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49725631 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78540765 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92842248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33884521 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121030656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚303³óÃü'†ŠN B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚584329616444538 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589521661160448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545975664502272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171356641 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92815882 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589521661160448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78540765 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12421530418688 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚598549431707648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9011933607558 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚545975785532928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3699 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12421460591104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575321316927840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9008324944666 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83609487 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚598545794641920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92815882 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49725631 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78540765 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92842248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33884521 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121030656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚303 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚584333250535398 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9024159873024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171356641 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49725636 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171383013 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚589521634768896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589521634768896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171383013 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¯/ïÍ–2‡- _wandbruntime‚6736  _runtime‚6736 ÅTˆÎ 0loss/FBt1/assignment_loss‚0.2064582374451592 .loss/FRt1/detection_loss‚1.753342050861313 .loss/FRt3/detection_loss‚2.953830757311412 epoch‚3 1loss/SRqqt1/detection_loss‚1.0492204816173052 0loss/SRqqt2/detection_loss‚3.060960807970592 .loss/FRt4/assignment_loss‚4.20140108695397 /loss/FRt4/detection_loss‚2.5593409905066857 1loss/SRqqt3/detection_loss‚2.5522187146647224 /loss/FBt2/detection_loss‚0.6396741349915308 %loss/total_loss‚47.75438756963615 2loss/SRqqt2/assignment_loss‚2.3848451993295123 0loss/FBt2/assignment_loss‚0.1958853741698107 $ _timestamp‚1.788476530075747e+09 1loss/FBt4/assignment_loss‚0.28755477375544947 .loss/FBt3/detection_loss‚1.769811733563741 .loss/FBt4/detection_loss‚2.913881778717041 trainer/global_step‚3699 /loss/FRt2/assignment_loss‚2.102322072424787 2loss/SRqqt1/assignment_loss‚1.1382789272843425 /loss/FBt3/assignment_loss‚0.789888775422393 1loss/SRqqt4/detection_loss‚2.1860410537038533 /loss/FBt1/detection_loss‚2.6249583851207388 _step‚304 0loss/FRt1/assignment_loss‚1.9570275711662628 /loss/FRt2/detection_loss‚2.2526193647942643 _runtime‚6736.831090146 2loss/SRqqt3/assignment_loss‚1.6127157827903484 1loss/SRqqt4/assignment_loss‚2.022998181836946 /loss/FRt3/assignment_loss‚4.539111333233969Ž­ Y‰Ó $ _timestamp‚1.788476530075747e+09 1loss/FBt4/assignment_loss‚0.28755477375544947 .loss/FBt3/detection_loss‚1.769811733563741 .loss/FBt4/detection_loss‚2.913881778717041 trainer/global_step‚3699 /loss/FRt2/assignment_loss‚2.102322072424787 2loss/SRqqt1/assignment_loss‚1.1382789272843425 /loss/FBt3/assignment_loss‚0.789888775422393 1loss/SRqqt4/detection_loss‚2.1860410537038533 /loss/FBt1/detection_loss‚2.6249583851207388 _step‚304 0loss/FRt1/assignment_loss‚1.9570275711662628 /loss/FRt2/detection_loss‚2.2526193647942643 _runtime‚6736.831090146 2loss/SRqqt3/assignment_loss‚1.6127157827903484 1loss/SRqqt4/assignment_loss‚2.022998181836946 /loss/FRt3/assignment_loss‚4.539111333233969 0loss/FBt1/assignment_loss‚0.2064582374451592 .loss/FRt1/detection_loss‚1.753342050861313 .loss/FRt3/detection_loss‚2.953830757311412 epoch‚3 1loss/SRqqt1/detection_loss‚1.0492204816173052 0loss/SRqqt2/detection_loss‚3.060960807970592 .loss/FRt4/assignment_loss‚4.20140108695397 /loss/FRt4/detection_loss‚2.5593409905066857 1loss/SRqqt3/detection_loss‚2.5522187146647224 /loss/FBt2/detection_loss‚0.6396741349915308 %loss/total_loss‚47.75438756963615 2loss/SRqqt2/assignment_loss‚2.3848451993295123 0loss/FBt2/assignment_loss‚0.1958853741698107°¯ß÷›Š‚(: ÿøçÔñçØx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚256.255' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.251000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð²ªx‹‚(:í ÿøçÔÜìª{ cpu‚25.154765664977933 network.sent‚ 1.002457e+07 disk.drbd1260.out‚ 0.94140625& disk.nvme0n1.out‚163101.7138671875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 2.126227e+06! disk.nvme0c0n1.in‚ 17671.28125 disk.dm-1.out‚ 204.6328125 disk.sda.in‚0 disk.sda3.out‚ 204.6328125 disk.drbd1260.in‚0 disk.sda1.in‚0* proc.memory.percent‚19.920969009399414) disk./.usagePercent‚78.07373961141907% disk./.usageGB‚1163.3551406860352 disk.dm-0.in‚ 17671.28125 disk.dm-1.in‚0 disk.sda.out‚ 204.6328125 disk.sda3.in‚0( disk.nvme0c0n1.out‚163101.7138671875# disk.dm-0.out‚163101.7138671875 disk.nvme0n1.in‚ 17671.28125 disk.sda1.out‚0$ memory_percent‚68.27726364135742' proc.memory.availableMB‚ 5197.453125" proc.memory.rssMB‚ 3263.8515625Œ— +~Œ‚(:ó ŽùçÔ€èx network.sent‚ 1.0131928e+07) disk./.usagePercent‚78.07295418030402# disk.nvme0c0n1.in‚ 17789.6171875 disk.sda.out‚ 205.109375! disk.nvme0n1.in‚ 17789.6171875& disk.nvme0n1.out‚163630.7685546875 cpu‚25.154847505665195% disk./.usageGB‚1163.3434371948242( disk.nvme0c0n1.out‚163630.7685546875 disk.drbd1260.out‚ 0.94140625 disk.sda1.out‚0$ memory_percent‚68.27776432037354" proc.memory.rssMB‚ 3263.8515625 disk.sda3.in‚0 network.recv‚ 2.131608e+06# disk.dm-0.out‚163630.7685546875 disk.dm-1.out‚ 205.109375 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 17789.6171875 disk.dm-1.in‚0 disk.sda3.out‚ 205.109375 disk.sda2.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 5197.37109375* proc.memory.percent‚19.920969009399414îR㈑‚(:† ŽùçÔµ¤Ðy gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚173.533' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë‹”W€Ž‚(:õ ùçÔøÔÿx( disk./.usagePercent‚78.0729593004547 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 205.4453125" proc.memory.rssMB‚ 3263.8515625* proc.memory.percent‚19.920969009399414 cpu‚25.15523553656639% disk./.usageGB‚1163.3435134887695' disk.nvme0n1.out‚164015.77197265625 disk.sda1.out‚0) disk.nvme0c0n1.out‚164015.77197265625$ memory_percent‚68.27797889709473 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 17789.6171875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.9453125( proc.memory.availableMB‚ 5197.3359375 network.sent‚ 1.0138096e+07 network.recv‚ 2.136071e+06! disk.nvme0n1.in‚ 17789.6171875# disk.nvme0c0n1.in‚ 17789.6171875$ disk.dm-0.out‚164015.77197265625 disk.dm-1.out‚ 205.4453125 disk.sda.in‚0 disk.sda.out‚ 205.4453125±ˆž‚(:’ ùçÔ”ê¼y gpu.0.gpu‚60.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚213.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u~(‚(:’ ¬ùçÔÿå¹x gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚283.403' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.680600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;”õ•†‘‚(:û ¬ùçÔ—º¢y) proc.memory.availableMB‚ 5197.31640625" proc.memory.rssMB‚ 3263.8515625 network.recv‚ 2.139595e+06) disk./.usagePercent‚78.06781738912949 disk.dm-1.out‚ 205.81640625 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 17789.6171875 disk.dm-0.in‚ 17789.6171875 disk.sda3.in‚0! disk.nvme0n1.in‚ 17789.6171875* proc.memory.percent‚19.920969009399414 cpu‚25.155076807277922 proc.cpu.threads‚17% disk./.usageGB‚1163.2668952941895$ disk.dm-0.out‚164304.45556640625 disk.sda3.out‚ 205.81640625 disk.drbd1260.in‚0$ memory_percent‚68.27809810638428 network.sent‚ 1.0145202e+07) disk.nvme0c0n1.out‚164304.45556640625 disk.dm-1.in‚0 disk.sda.out‚ 205.81640625 disk.drbd1260.out‚ 0.9453125' disk.nvme0n1.out‚164304.45556640625“$,V2’- _wandbruntime‚6806  _runtime‚6806¢!¤<Ç“Á & lr-AdamW/pg1‚0.0005680670313165054 & lr-AdamW/pg2‚0.0005680670313165054 trainer/global_step‚3749 % _timestamp‚1.7884765996583598e+09 _runtime‚6806.413922671 _step‚305ÑÐSÌ”Æ & lr-AdamW/pg2‚0.0005680670313165054 trainer/global_step‚3749 % _timestamp‚1.7884765996583598e+09 _runtime‚6806.413922671 _step‚305 & lr-AdamW/pg1‚0.0005680670313165054±håFL2•- _wandbruntime‚6806  _runtime‚6806ÜÛa[$–üO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6806.415103977 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9129856059572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚606202660102656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84673199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591813671712424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94049357 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94023017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540379432971776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50353081 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 _step‚306 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79541535 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9145898549248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚606202660102656 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9142287888384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚597056761553408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84672487 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50353076 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚591817305803076 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94049357 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540379484174848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚606199023050240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884765996594949e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9145898549248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552964538778624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚582687424315536 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12584982665728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reser&Ry§ved_bytes.large_pool.allocated‚ 28808577024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34320118 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79541541 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34319411 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400hR (—P PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79541535 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12585054603776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚582687449743504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173564552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79541541 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚552964415637504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12584982665728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34320118 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79541541 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34319411 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6806.415103977 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9129856059572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚606202660102656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84673199 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚591813671712424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94049357 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94023017 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540379432971776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50353081 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 _step‚306 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79541535 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9145898549248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚606202660102656 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9142287888384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚597056761553408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84672487 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50353076 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚591817305803076 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94049357 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540379484174848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚707 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚606199023050240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884765996594949e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9145898549248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552964538778624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚582687424315536 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173635915 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚582837778984640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚597210512720896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34328241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173609543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚6807.971814891 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84694122 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9132262363066 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 _step‚307 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84694814 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94047584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591966407256846 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚597210539112448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚606358848156672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚597210512720896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79561965 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50365886 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9148309044224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79561959 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚582837753556672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9148309044224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94047584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚553107025969664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591970041347706 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚606355211090944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ŒÒ ]'š‹N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚597210539112448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚606358848156672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚597210512720896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79561965 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50365886 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9148309044224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79561959 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚582837753556672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9148309044224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94047584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚553107025969664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591970041347706 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚606355211090944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚606358848156672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚597210539112448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173635915 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚582837778984640 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚597210512720896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34328241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173609543 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚6807.971814891 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84694122 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9132262363066 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 _step‚307 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84694814 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94047584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591966407256846 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0³Nmój2›- _wandbruntime‚6807  _runtime‚68078žUYœÓ 1loss/FBt2/assignment_loss‚0.12802165800386173 .loss/FRt2/detection_loss‚2.026657559139573 .loss/FBt1/detection_loss‚3.409904032945633 epoch‚3 2loss/SRqqt3/assignment_loss‚1.9922197430298245 1loss/FBt4/assignment_loss‚0.20585843052518052 /loss/FBt3/detection_loss‚2.6876682681696757 %loss/total_loss‚51.55435638390449 trainer/global_step‚3749 1loss/SRqqt4/assignment_loss‚2.278859986861547 0loss/FBt1/assignment_loss‚0.3053854964673519 0loss/SRqqt2/detection_loss‚3.050578291599567 0loss/SRqqt3/detection_loss‚2.236788984002738 $ _timestamp‚1.788476601218422e+09 2loss/SRqqt2/assignment_loss‚2.6233383142031155 /loss/FBt2/detection_loss‚0.8436908127328793 0loss/FRt2/assignment_loss‚2.9513320132442145 2loss/SRqqt1/assignment_loss‚1.2534301816403404 0loss/FRt1/assignment_loss‚2.6510112850289596 1loss/FBt3/assignment_loss‚0.44724505665570696 _runtime‚6807.974056947 /loss/FRt3/assignment_loss‚4.880767207879287 .loss/FRt3/detection_loss‚3.270755655490435 /loss/FBt4/detection_loss‚3.0313960634745083 _step‚308 0loss/FRt4/assignment_loss‚3.4426879666068335 /loss/FRt1/detection_loss‚1.5720068635273252 .loss/FRt4/detection_loss‚2.757856755545645 1loss/SRqqt1/detection_loss‚0.9970544103895669 1loss/SRqqt4/detection_loss‚2.5098413467407226µÓð@Ø 2loss/SRqqt3/assignment_loss‚1.9922197430298245 1loÉ _ss/FBt4/assignment_loss‚0.20585843052518052 /loss/FBt3/detection_loss‚2.6876682681696757 %loss/total_loss‚51.55435638390449 trainer/global_step‚3749 1loss/SRqqt4/assignment_loss‚2.278859986861547 0loss/FBt1/assignment_loss‚0.3053854964673519 0loss/SRqqt2/detection_loss‚3.050578291599567 0loss/SRqqt3/detection_loss‚2.236788984002738 $ _timestamp‚1.788476601218422e+09 2loss/SRqqt2/assignment_loss‚2.6233383142031155 /loss/FBt2/detection_loss‚0.8436908127328793 0loss/FRt2/assignment_loss‚2.9513320132442145 2loss/SRqqt1/assignment_loss‚1.2534301816403404 0loss/FRt1/assignment_loss‚2.6510112850289596 1loss/FBt3/assignment_loss‚0.44724505665570696 _runtime‚6807.974056947 /loss/FRt3/assignment_loss‚4.880767207879287 .loss/FRt3/detection_loss‚3.270755655490435 /loss/FBt4/detection_loss‚3.0313960634745083 _step‚308 0loss/FRt4/assignment_loss‚3.4426879666068335 /loss/FRt1/detection_loss‚1.5720068635273252 .loss/FRt4/detection_loss‚2.757856755545645 1loss/SRqqt1/detection_loss‚0.9970544103895669 1loss/SRqqt4/detection_loss‚2.5098413467407226 1loss/FBt2/assignment_loss‚0.12802165800386173 .loss/FRt2/detection_loss‚2.026657559139573 .loss/FBt1/detection_loss‚3.409904032945633 epoch‚3´Ë벆vž‚(:ë »ùçÔ˜°Ýx disk.sda.in‚0 network.sent‚ 1.014968e+07 disk.drbd1260.out‚ 0.94921875 disk.sda2.out‚0& disk.nvme0c0n1.out‚164563.49609375 disk.dm-1.in‚0& proc.memory.availableMB‚ 5197.21875 cpu‚25.155033753491644 proc.cpu.threads‚17) disk./.usagePercent‚78.07838461212032 disk.sda3.in‚0! disk.nvme0n1.in‚ 17789.6171875$ disk.nvme0n1.out‚164563.49609375 disk.sda2.in‚0 disk.sda.out‚ 206.3046875$ memory_percent‚68.27869415283203" proc.memory.rssMB‚ 3263.8515625% disk./.usageGB‚1163.4243545532227 disk.sda3.out‚ 206.3046875 disk.drbd1260.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 17789.6171875! disk.dm-0.out‚164563.49609375 disk.dm-1.out‚ 206.3046875* proc.memory.percent‚19.920969009399414 network.recv‚ 2.14469e+06 disk.sda1.in‚0 disk.dm-0.in‚ 17789.6171875¢¥hžŸ‚(:… »ùçÔ“ úy gpu.0.gpu‚2.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚127.309' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4618 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uûR”‘ ‚(:† ÊùçÔ‰æšy gpu.0.gpu‚52.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚195.996' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.1992 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ølÇŸ{¡‚(:ð ÊùçÔÇ·èy cpu‚25.155339866135684 proc.cpu.threads‚17 network.sent‚ 1.0257307e+07 network.recv‚ 2.150269e+06) disk./.usagePercent‚78.07680965376967! disk.nvme0n1.in‚ 17926.5546875 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚68.27869415283203$ disk.nvme0n1.out‚164756.06640625 disk.sda1.out‚0& disk.nvme0c0n1.out‚164756.06640625 disk.dm-0.in‚ 17926.5546875! disk.dm-0.out‚164756.06640625 disk.dm-1.out‚ 206.72265625 disk.sda3.out‚ 206.72265625 disk.drbd1260.in‚0% disk./.usageGB‚1163.4008865356445 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1260.out‚ 0.94921875" proc.memory.rssMB‚ 3263.8515625* proc.memory.percent‚19.920969009399414 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 17926.5546875 disk.dm-1.in‚0 disk.sda.out‚ 206.72265625& proc.memory.availableMB‚ 5197.21875©#nu¢‚(:ê ÙùçÔòôÉx disk.sda1.out‚0# memory_percent‚95.6568717956543 cpu‚25.15537377677948% disk./.usageGB‚1163.4053993225098) disk.nvme0c0n1.out‚165000.81298828125 disk.dm-1.out‚ 207.4453125 disk.sda.out‚ 207.4453125" proc.memory.rssMB‚ 3263.8515625* proc.memory.percent‚19.920969009399414 proc.cpu.threads‚17 network.recv‚ 2.1549e+06) disk./.usagePercent‚78.07711251068268 disk.dm-0.in‚ 17926.84375 disk.drbd1260.out‚0.953125 disk.nvme0n1.in‚ 17926.84375& proc.memory.availableMB‚ 711.578125! disk.nvme0c0n1.in‚ 17926.84375$ disk.dm-0.out‚165000.81298828125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 1.0263461e+07 disk.sda3.out‚ 207.4453125 disk.drbd1260.in‚0' disk.nvme0n1.out‚165000.81298828125ýÇ‹=‘£‚(:† ÙùçÔžã¨y gpu.0.gpu‚46.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚207.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ŒÑªs¤‚(:è èùçÔ¼Äáy disk.sda1.out‚0$ memory_percent‚95.65632343292236$ disk./.usageGB‚1163.428867340088 disk.dm-1.out‚ 207.84375 disk.sda.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3263.8515625* proc.memory.percent‚19.920969009399414 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 17926.84375) disk.nvme0c0n1.out‚165529.80517578125$ disk.dm-0.out‚165529.80517578125 disk.sda.out‚ 207.84375 disk.drbd1260.in‚0' disk.nvme0n1.out‚165529.80517578125 disk.sda1.in‚0 network.recv‚ 2.158027e+06 disk.dm-1.in‚0 disk.drbd1260.out‚0.953125 disk.nvme0n1.in‚ 17926.84375( proc.memory.availableMB‚ 711.66796875 cpu‚25.155532950770784 network.sent‚ 1.027031e+07) disk./.usagePercent‚78.07868746903333 disk.dm-0.in‚ 17926.84375 disk.sda3.in‚0 disk.sda3.out‚ 207.84375 disk.sda2.in‚0«DTŽ¥‚(:’ èùçÔ’Æ€z gpu.0.gpu‚57.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚215.283' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.056599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ª¸P¦‚(:„ ÷ùçÔª£¢{ gpu.0.gpu‚39.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06u`sx§‚(:í ÷ùçÔ‰»Î|' disk.nvme0c0n1.out‚165948.763671875 disk.dm-0.in‚ 17926.84375 disk.drbd1260.in‚0( proc.memory.availableMB‚ 711.66796875" proc.memory.rssMB‚ 3263.8515625 cpu‚25.15578706194817 proc.cpu.threads‚17% disk./.usageGB‚1163.4913291931152! disk.nvme0c0n1.in‚ 17926.84375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 208.39453125 disk.sda1.out‚0$ memory_percent‚95.65632343292236* proc.memory.percent‚19.920969009399414 network.sent‚ 1.027483e+07 disk.sda.out‚ 208.39453125 disk.drbd1260.out‚ 0.95703125% disk.nvme0n1.out‚165948.763671875 network.recv‚ 2.163173e+06" disk.dm-0.out‚165948.763671875 disk.dm-1.out‚ 208.39453125 disk.sda3.in‚0 disk.nvme0n1.in‚ 17926.84375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0) disk./.usagePercent‚78.08287933639897ö‘ꨂ(:ô †úçÔùîÓy$ memory_percent‚95.65632343292236 network.sent‚ 1.0279769e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 208.72265625 disk.sda.out‚ 208.72265625 disk.sda3.out‚ 208.72265625% disk.nvme0n1.out‚166188.275390625 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 18063.3828125" disk.dm-0.out‚166188.275390625 disk.sda3.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1163.5540771484375! disk.nvme0n1.in‚ 18063.3828125 disk.dm-0.in‚ 18063.3828125 disk.sda.in‚0( proc.memory.availableMB‚ 711.66796875" proc.memory.rssMB‚ 3263.8515625* proc.memory.percent‚19.920969009399414 cpu‚25.156047215007135 network.recv‚ 2.166836e+06( disk./.usagePercent‚78.0870904043297' disk.nvme0c0n1.out‚166188.275390625 disk.drbd1260.out‚ 0.957031250 »Cœ©‚(:‘ †úçÔ¿æz gpu.0.gpu‚14.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.476' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.895199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_.k=xª‚(:í •úçÔ¨Ù³y disk.sda3.out‚ 209.15625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.9609375 network.recv‚ 2.171432e+06$ disk./.usageGB‚1163.678810119629! disk.nvme0n1.in‚ 18063.3828125 disk.sda1.in‚0 disk.dm-1.out‚ 209.15625" proc.memory.rssMB‚ 3263.8515625* proc.memory.percent‚19.920969009399414 disk.sda2.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚166641.14892578125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 209.15625 disk.sda3.in‚0$ memory_percent‚95.65634727478027# disk.nvme0c0n1.in‚ 18063.3828125 disk.dm-0.in‚ 18063.3828125' proc.memory.availableMB‚ 711.6640625 cpu‚25.15621472011236 proc.cpu.threads‚17 network.sent‚ 1.028781e+07) disk./.usagePercent‚78.09546133868429' disk.nvme0n1.out‚166641.14892578125 disk.sda1.out‚0) disk.nvme0c0n1.out‚166641.14892578125SÑ«‚(:† •úçÔ»ÔÓy gpu.0.gpu‚27.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.732' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä"O2¬- _wandbruntime‚6902  _runtime‚6902‰|½Æ­À trainer/global_step‚3799 $ _timestamp‚1.788476695965021e+09 _runtime‚6902.722249697 _step‚309 & lr-AdamW/pg1‚0.0005648924637549882 & lr-AdamW/pg2‚0.0005648924637549882J¯kÝˮŠ& lr-AdamW/pg2‚0.0005648924637549882 trainer/global_step‚3799 $ _timestamp‚1.788476695965021e+09 _runtime‚6902.722249697 _step‚309 & lr-AdamW/pg1‚0.0005648924637549882µ`“»2¯- _wandbruntime‚6902  _runtime‚6902þ(Z(°üO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50993326 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚590203911800304 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚614008439499264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9266437046272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95281059 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50993331 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604745639505408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚599454096615384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚599450462524732 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80562735 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95281059 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175843800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80562741 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604745613113856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9250184815080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95254719 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 _step‚310 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9266437046272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚3799 _runtime‚6902.727329995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80562741 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604745613113856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175817454 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚614012076551680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547343952421376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12751708155904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85756434 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12751634120704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚547344003624448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9246576152396 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85755727 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚614012076551680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560095586542080 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚614012076551680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560095586542080 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚590203911800304 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚614008439499264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9266437046272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95281059 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50993331 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604745639505408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚599454096615384 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚599450462524732 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80562735 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95281059 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175843800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80562741 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604745613113856 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9250184815080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95254719 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 _step‚310 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9266437046272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚3799 _runtime‚6902.727329995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80562741 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604745613113856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0¶ƒŽ2²- _wandbruntime‚6904  _runtime‚6904qG¤b¾³†N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175862445 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80583159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604899390672896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95305652 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚371ûÄC®N"9 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560238328041472 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85777376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚590354215613472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80583165 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚599606832160014 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9268847541248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚547483242810368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80583165 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34771931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34771245 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175862445 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9252591118574 % _timestamp‚1.7884766976336765e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175888817 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85778067 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12754960006144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604899390672896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80583159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51006136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚614168264605696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265236867072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560238202816512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604899417064448 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9268847541248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚6904.392379653 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚614168264605696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175888817 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95279286 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9248982455682 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604899417064448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚599603198069154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 92652368670725/Ö'´‹N F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12754960006144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604899390672896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80583159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51006136 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚614168264605696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265236867072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560238202816512 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604899417064448 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9268847541248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚6904.392379653 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚614168264605696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175888817 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95279286 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9248982455682 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604899417064448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚599603198069154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9265236867072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175862445 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80583159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604899390672896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95305652 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560238328041472 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85777376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚590354215613472 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80583165 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚599606832160014 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9268847541248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚547483242810368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3799 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80583165 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34771931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34771245 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175862445 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9252591118574 % _timestamp‚1.7884766976336765e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175888817 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85778067 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86837298 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621817855948288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12917905344000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607090887427692 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621817855948288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9386975543296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚612434491065856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51633581 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9383364882432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35203717 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚554308523074048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597720373857104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81583941 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚554308471870976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9370513570588 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _step‚314 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96486421 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816šÇ-¾(ÉP YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12917977282048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚612434517457408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚567226500356096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178096702 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86837298 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621817855948288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12917905344000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607090887427692 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621817855948288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9386975543296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚612434491065856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51633581 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9383364882432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35203717 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚554308523074048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597720373857104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81583941 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚554308471870976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9370513570588 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _step‚314 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96486421 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816º1ÆØ92Ê- _wandbruntime‚6994  _runtime‚6994W£Wn 'ˆN WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚612588268624896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚612588295016448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12921226558976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86858820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚612588268624896 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚554447762259968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚567369111946752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9385775364096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35211750 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567368988818944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607243622972322 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚315 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81604359 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96537354 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96510988 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚554447813463040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86858131 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51646386 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178141719 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178115347 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178141719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 % _timestamp‚1.7884767882067869e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81604365 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178115347 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81604365 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚612588295016448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81604359 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35212434 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9369311211190 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597870677670272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96510988 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621977681054720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389386038272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9389386038272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51646381 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9372919874082 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552°<3Ì‹N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178141719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 % _timestamp‚1.7884767882067869e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81604365 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178115347 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81604365 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚612588295016448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81604359 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35212434 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9369311211190 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597870677670272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96510988 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚621977681054720 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389386038272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9389386038272 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51646381 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9372919874082 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚612588268624896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚612588295016448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12921226558976 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86858820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 I@DeviÅ^„^ÞceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚612588268624896 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚554447762259968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚567369111946752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9385775364096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35211750 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567368988818944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607243622972322 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚315 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81604359 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96537354 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96510988 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚554447813463040 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86858131 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51646386 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178141719 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178115347 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3849»Ê²#2Í- _wandbruntime‚6994  _runtime‚6994¤ÀÂçTÎÎ /loss/FBt3/detection_loss‚2.8572658679702063 epoch‚3 2loss/SRqqt3/assignment_loss‚1.6222328995664914 1loss/FBt2/assignment_loss‚0.07644812877822003 0loss/FBt3/assignment_loss‚0.5200369288332612 1loss/SRqqt3/detection_loss‚2.2801762501398724 .loss/FBt2/detection_loss‚0.834190111341221 /loss/FBt4/assignment_loss‚2.390348519449948 1loss/SRqqt1/detection_loss‚1.0934894969196696 /loss/SRqqt2/detection_loss‚3.77481260150671 _runtime‚6994.973432239 0loss/FBt1/assignment_loss‚0.0697556722925204 1loss/SRqqt4/detection_loss‚2.3477960756846836 .loss/FBt4/detection_loss‚2.796558678150177 _step‚316 /loss/FRt3/assignment_loss‚4.160285678776828 1loss/SRqqt2/assignment_loss‚2.214632198214531 /loss/FRt2/detection_loss‚1.9337776482105256 0loss/FRt4/assignment_loss‚3.5457365530909914 1loss/SRqqt1/assignment_loss‚1.517379176884628 /loss/FBt1/detection_loss‚2.5055273175239563 &loss/total_loss‚51.052856266737976 trainer/global_step‚3849 2loss/SRqqt4/assignment_loss‚1.9853551621415786 /loss/FRt4/detection_loss‚2.7120495384389702 % _timestamp‚1.7884767882166076e+09 /loss/FRt1/assignment_loss‚2.046766998275878 /loss/FRt2/assignment_loss‚2.838547180891037 .loss/FRt1/detection_loss‚1.706184842037478 -loss/FRt3/detection_loss‚3.22350274161859‹³¶YÏÓ /loss/FRt3/assignment_loss‚4.160285678776828 1loss/SRqqt2/assignment_loss‚2.214632198214531 /loss/FRt2/detection_loss‚1.9337776482105256 0loss/FRt4/assignment_loss‚3.5457365530909914 1loss/SRqqt1/assignment_loss‚1.517379176884628 /loss/FBt1/detection_loss‚2.5055273175239563 &loss/total_loss‚51.052856266737976 trainer/global_step‚3849 2loss/SRqqt4/assignment_loss‚1.9853551621415786 /loss/FRt4/detection_loss‚2.7120495384389702 % _timestamp‚1.7884767882166076e+09 /loss/FRt1/assignment_loss‚2.046766998275878 /loss/FRt2/assignment_loss‚2.838547180891037 .loss/FRt1/detection_loss‚1.706184842037478 -loss/FRt3/detection_loss‚3.22350274161859 /loss/FBt3/detection_loss‚2.8572658679702063 epoch‚3 2loss/SRqqt3/assignment_loss‚1.6222328995664914 1loss/FBt2/assignment_loss‚0.07644812877822003 0loss/FBt3/assignment_loss‚0.5200369288332612 1loss/SRqqt3/detection_loss‚2.2801762501398724 .loss/FBt2/detection_loss‚0.834190111341221 /loss/FBt4/assignment_loss‚2.390348519449948 1loss/SRqqt1/detection_loss‚1.0934894969196696 /loss/SRqqt2/detection_loss‚3.77481260150671 _runtime‚6994.973432239 0loss/FBt1/assignment_loss‚0.0697556722925204 1loss/SRqqt4/detection_loss‚2.3477960756846836 .loss/FBt4/detection_loss‚2.796558678150177 _step‚316¼Yã ù…Ђ(:ú þúçÔ“¯¡y disk.sda1.in‚0$ disk.nvme0c0n1.in‚18184.87890625 disk.dm-1.out‚ 212.3359375 disk.sda.out‚ 212.3359375 disk.sda3.out‚ 212.3359375' proc.memory.availableMB‚ 712.0859375 proc.cpu.threads‚17 network.recv‚ 2.200337e+06 disk.sda2.in‚0) disk.nvme0c0n1.out‚168982.20361328125$ disk.dm-0.out‚168982.20361328125 disk.sda3.in‚0' disk.nvme0n1.out‚168982.20361328125$ memory_percent‚95.65377235412598* proc.memory.percent‚19.921374320983887) disk./.usagePercent‚78.04401483262596 disk.sda1.out‚0 disk.sda.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚18184.87890625 cpu‚25.15739127361626 disk.dm-1.in‚0 network.sent‚ 1.0425961e+07% disk./.usageGB‚1162.9122200012207 disk.sda2.out‚0 disk.dm-0.in‚18184.87890625 disk.drbd1260.out‚ 0.97265625# proc.memory.rssMB‚ 3263.91796875Ì’Ñ‚(:‡ þúçÔÞžÚy gpu.0.gpu‚65.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.393' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.8786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿtª8œÒ‚(:‘ ûçÔÄÝñw gpu.0.gpu‚19.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.577' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.315400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÿ”nÓ‚(:ã ûçÔ—ÛÉy network.recv‚ 2.207267e+06" disk.nvme0n1.in‚18364.36328125 disk.sda2.out‚0 disk.dm-1.out‚ 212.71875 disk.sda3.out‚ 212.71875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.9765625 proc.memory.availableMB‚712 disk.sda2.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚18364.36328125 disk.dm-0.in‚18364.36328125 disk.sda.in‚0# proc.memory.rssMB‚ 3263.91796875* proc.memory.percent‚19.921374320983887 cpu‚25.15761934450761 network.sent‚ 1.0534711e+07) disk./.usagePercent‚78.04401841673145% disk./.usageGB‚1162.9122734069824 disk.sda1.in‚0# disk.dm-0.out‚169210.8974609375 memory_percent‚ 95.654296875& disk.nvme0n1.out‚169210.8974609375( disk.nvme0c0n1.out‚169210.8974609375 disk.dm-1.in‚0 disk.sda.out‚ 212.71875 disk.sda3.in‚0 proc.cpu.threads‚17zf»ïÔ‚(:ö œûçÔ»®y disk.dm-1.in‚0 disk.dm-1.out‚ 213.1953125' proc.memory.availableMB‚ 712.0078125# proc.memory.rssMB‚ 3263.91796875* proc.memory.percent‚19.921374320983887 disk.sda2.out‚0 disk.sda3.out‚ 213.1953125 disk.sda2.in‚0 disk.dm-0.in‚18364.36328125$ memory_percent‚95.65424919128418 cpu‚25.157817009197952) disk./.usagePercent‚78.04402353688212% disk./.usageGB‚1162.9123497009277 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 2.21052e+06 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.9765625" disk.nvme0n1.in‚18364.36328125& disk.nvme0n1.out‚169676.7958984375$ disk.nvme0c0n1.in‚18364.36328125( disk.nvme0c0n1.out‚169676.7958984375 network.sent‚ 1.0539554e+07 disk.sda.out‚ 213.1953125# disk.dm-0.out‚169676.7958984375={ç#’Õ‚(:‡ œûçԜʪy gpu.0.gpu‚94.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚116.629' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÞøˉÖ‚(:þ «ûçÔü¤¬}# proc.memory.rssMB‚ 3263.91796875 cpu‚25.158049490541284 proc.cpu.threads‚17 disk.sda3.out‚ 213.51171875$ disk.nvme0c0n1.in‚18364.36328125 disk.dm-0.in‚18364.36328125 disk.dm-1.in‚0 disk.sda.out‚ 213.51171875 disk.drbd1260.out‚ 0.98046875 disk.sda1.in‚0$ memory_percent‚95.65427303314209 network.recv‚ 2.215382e+06$ disk./.usageGB‚1162.888843536377) disk.nvme0c0n1.out‚170083.79638671875 disk.sda3.in‚0' disk.nvme0n1.out‚170083.79638671875 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 712.00390625) disk./.usagePercent‚78.04244601845613$ disk.dm-0.out‚170083.79638671875 disk.dm-1.out‚ 213.51171875 disk.sda.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚18364.36328125* proc.memory.percent‚19.921374320983887 network.sent‚ 1.0545946e+07 disk.sda1.out‚0#ýŒׂ(:„ «ûçÔ»÷ƒ~ gpu.0.gpu‚90.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.45' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.69 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãÄñƒØ‚(:ø ºûçÔœ†{% disk./.usageGB‚1162.8888893127441$ disk.nvme0c0n1.in‚18364.36328125$ disk.dm-0.out‚170466.48388671875 disk.dm-1.in‚0 disk.sda.out‚ 214.015625 disk.sda3.in‚0# proc.memory.rssMB‚ 3263.91796875 proc.cpu.threads‚17 network.sent‚ 1.0550932e+07) disk./.usagePercent‚78.04244909054655 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.98046875 disk.sda2.in‚0 disk.dm-1.out‚ 214.015625 disk.sda3.out‚ 214.015625' proc.memory.availableMB‚ 712.0078125 network.recv‚ 2.218748e+06' disk.nvme0n1.out‚170466.48388671875 disk.sda2.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚170466.48388671875 disk.sda.in‚0$ memory_percent‚95.65424919128418* proc.memory.percent‚19.921374320983887" disk.nvme0n1.in‚18364.36328125 disk.sda1.in‚0 disk.dm-0.in‚18364.36328125 cpu‚25.158224068647645˜sڜق(:‘ ºûçÔíÊ•{ gpu.0.gpu‚40.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚120.918' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.183600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷n‘Ú‚(:† ÉûçÔàþ–y gpu.0.gpu‚49.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚171.661' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3322 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0føƒ€yÛ‚(:î ÉûçÔ´û¹z( disk./.usagePercent‚78.0440240488972& disk.nvme0c0n1.out‚170812.32421875 disk.dm-1.out‚ 214.4453125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.984375$ disk.nvme0n1.out‚170812.32421875 disk.sda2.in‚0% disk./.usageGB‚1162.9123573303223 disk.sda3.out‚ 214.4453125" disk.nvme0n1.in‚18503.43359375 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.91796875 proc.cpu.threads‚17 network.sent‚ 1.0558888e+07 network.recv‚ 2.223387e+06 disk.sda.in‚0 disk.sda.out‚ 214.4453125$ memory_percent‚95.65427303314209* proc.memory.percent‚19.921374320983887 cpu‚25.1583506623623$ disk.nvme0c0n1.in‚18503.43359375 disk.dm-0.in‚18503.43359375! disk.dm-0.out‚170812.32421875 disk.dm-1.in‚0( proc.memory.availableMB‚ 712.00390625•Èø2Ü- _wandbruntime‚7089  _runtime‚7089vD¬ÀÂݼ $ lr-AdamW/pg1‚0.00055839459985778 $ lr-AdamW/pg2‚0.00055839459985778 trainer/global_step‚3899 % _timestamp‚1.7884768827392511e+09 _runtime‚ 7089.49526565 _step‚317Ö¯BæÇÞÁ $ lr-AdamW/pg1‚0.00055839459985778 $ lr-AdamW/pg2‚0.00055839459985778 trainer/global_step‚3899 % _timestamp‚1.7884768827392511e+09 _runtime‚ 7089.49526565 _step‚317½Ῐ²2ß- _wandbruntime‚7089  _runtime‚7089v5Á½(àüO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚574356960846336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚318 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82605141 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚629627272397312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82605141 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚605236810485936 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35644090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884768827408314e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97744463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9503903379456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52273826 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚561273042523648 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13083969525760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚561272991320576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚7089.496425997 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚629627272397312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180349604 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35643385 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9487233663412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚629630909449728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13084043560960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97744463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97718123 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87917211 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚629627272397312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180349604 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35643385 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9487233663412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚629630909449728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13084043560960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97744463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97718123 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87917211 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35644090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884768827408314e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97744463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9503903379456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52273826 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚561273042523648 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13083969525760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚561272991320576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚7089.496425997¾*ÎĹ2â- _wandbruntime‚7090  _runtime‚7090jÞ;§ 'ã†N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180394621 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620277172968448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚629783460438016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82625565 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9506313861120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚614880413784630 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13087360477696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9509924535296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97742690 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180394621 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚629787097503744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35652223 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82625559 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180368249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97769056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9506313861120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚629787097503744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52286636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620277146576896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82625559 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87938854 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82625565 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9506313861120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚614880413784630 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13087360477696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9509924535296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97742690 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180394621 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚629787097503744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35652223 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82625559 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180368249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97769056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9506313861120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚629787097503744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52286636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620277146576896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82625559 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87938854 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¿Ñ®£2å- _wandbruntime‚7090  _runtime‚7090 IoYæÓ 0loss/FRt1/assignment_loss‚2.4402606268723805 0loss/FBt3/assignment_loss‚0.6594407364667859 1loss/FBt4/assignment_loss‚0.18209015465368641 1loss/SRqqt3/detection_loss‚2.6606914892792703 /loss/FBt1/detection_loss‚3.5207896743501936 .loss/FBt4/detection_loss‚2.802254811922709 0loss/FRt3/assignment_loss‚4.3671306272347765 /loss/FRt1/detection_loss‚1.6201400948436155 1loss/SRqqt4/detection_loss‚2.2998540788083464 /loss/FRt2/detection_loss‚1.7590243370893102 epoch‚3 $ _timestamp‚1.788476883752875e+09 _runtime‚ 7090.5081905 0loss/FBt2/assignment_loss‚0.1873713821578826 /loss/FRt3/detection_loss‚3.0743031402428946 0loss/SRqqt1/detection_loss‚1.163206260249603 1loss/SRqqt2/assignment_loss‚2.123186712153256 /loss/FBt2/detection_loss‚0.8029121880897799 0loss/FRt2/assignment_loss‚2.5345277823507786 0loss/FBt1/assignment_loss‚0.0299678203258996 /loss/FBt3/detection_loss‚2.3750158588091534 0loss/FRt4/assignment_loss‚3.2160742029999243 2loss/SRqqt3/assignment_loss‚2.1423439336940646 1loss/SRqqt2/detection_loss‚2.6505198385566473 _step‚320 2loss/SRqqt1/assignment_loss‚1.0626421943672149 2loss/SRqqt4/assignment_loss‚1.6767607446055155 .loss/FRt4/detection_loss‚2.890299854856549 %loss/total_loss‚48.24080854498024 trainer/global_step‚3899$cåe^çØ 0loss/FBt2/assignment_loss‚0.1873713821578826 /loss/FRt3/detection_loss‚3.0743031402428946 0loss/SRqqt1/detection_loss‚1.163206260249603 1loss/SRqqt2/assignment_loss‚2.123186712153256 /loss/FBt2/detection_loss‚0.8029121880897799 0loss/FRt2/assignment_loss‚2.5345277823507786 0loss/FBt1/assignment_loss‚0.0299678203258996 /loss/FBt3/detection_loss‚2.3750158588091534 0loss/FRt4/assignment_loss‚3.2160742029999243 2loss/SRqqt3/assignment_loss‚2.1423439336940646 1loss/SRqqt2/detection_loss‚2.6505198385566473 _step‚320 2loss/SRqqt1/assignment_loss‚1.0626421943672149 2loss/SRqqt4/assignment_loss‚1.6767607446055155 .loss/FRt4/detection_loss‚2.890299854856549 %loss/total_loss‚48.24080854498024 trainer/global_step‚3899 0loss/FRt1/assignment_loss‚2.4402606268723805 0loss/FBt3/assignment_loss‚0.6594407364667859 1loss/FBt4/assignment_loss‚0.18209015465368641 1loss/SRqqt3/detection_loss‚2.6606914892792703 /loss/FBt1/detection_loss‚3.5207896743501936 .loss/FBt4/detection_loss‚2.802254811922709 0loss/FRt3/assignment_loss‚4.3671306272347765 /loss/FRt1/detection_loss‚1.6201400948436155 1loss/SRqqt4/detection_loss‚2.2998540788083464 /loss/FRt2/detection_loss‚1.7590243370893102 epoch‚3 $ _timestamp‚1.788476883752875e+09 _runtime‚ 7090.5081905ÀFE8è‚(:… ØûçÔœ¹Î{ gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚204.611' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.9222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö'ÜÛxé‚(:í ØûçÔæ€ò} disk.sda2.in‚0$ memory_percent‚95.65598964691162$ disk.nvme0c0n1.in‚18503.43359375 disk.dm-1.out‚ 214.9609375 disk.sda.out‚ 214.9609375 disk.sda3.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 711.72265625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda3.out‚ 214.9609375# proc.memory.rssMB‚ 3263.91796875* proc.memory.percent‚19.921374320983887 cpu‚25.158551716665325( disk./.usagePercent‚78.0440240488972% disk.nvme0c0n1.out‚171062.8359375 disk.dm-0.out‚171062.8359375 disk.drbd1260.out‚0.984375" disk.nvme0n1.in‚18503.43359375# disk.nvme0n1.out‚171062.8359375 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 1.0561911e+07 network.recv‚ 2.226685e+06% disk./.usageGB‚1162.9123573303223 disk.dm-0.in‚18503.43359375 disk.sda.in‚0 disk.drbd1260.in‚0ŒÓm"{ê‚(:ð çûçÔøîÄx disk.sda.out‚ 215.46875 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚18503.43359375( disk.nvme0c0n1.out‚171409.8642578125 disk.dm-1.out‚ 215.46875 disk.drbd1260.out‚ 0.98828125 disk.sda2.out‚0$ memory_percent‚95.65446376800537# proc.memory.rssMB‚ 3263.91796875 network.sent‚ 1.067155e+07 disk.dm-0.in‚18503.43359375 disk.sda.in‚0( proc.memory.availableMB‚ 711.97265625 cpu‚25.1587806163889 proc.cpu.threads‚17 network.recv‚ 2.234625e+06 disk.sda1.in‚0 disk.sda3.out‚ 215.46875" disk.nvme0n1.in‚18503.43359375& disk.nvme0n1.out‚171409.8642578125 disk.sda2.in‚0* proc.memory.percent‚19.921374320983887) disk./.usagePercent‚78.04244909054655% disk./.usageGB‚1162.8888893127441# disk.dm-0.out‚171409.8642578125 disk.dm-1.in‚0/õ(¢‘ë‚(:† çûçÔ’ø·{ gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚202.683' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.5366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0döF¦›ì‚(: öûçÔŸÿ­‚ gpu.0.gpu‚4.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠŠúÔí‚(:ö öûçÔ˜°‘„$ memory_percent‚95.65448760986328# proc.memory.rssMB‚ 3263.91796875 proc.cpu.threads‚17% disk./.usageGB‚1162.9123573303223" disk.nvme0n1.in‚18503.43359375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.0679675e+07 disk.sda3.in‚0 disk.drbd1260.out‚ 0.98828125 disk.sda2.in‚0$ disk.nvme0c0n1.in‚18503.43359375 disk.dm-0.in‚18503.43359375 disk.sda3.out‚ 215.8984375& disk.nvme0n1.out‚171851.5673828125# disk.dm-0.out‚171851.5673828125 disk.dm-1.out‚ 215.8984375 disk.sda.out‚ 215.8984375% proc.memory.availableMB‚ 711.96875* proc.memory.percent‚19.921374320983887 cpu‚25.158989867955885 network.recv‚ 2.244708e+06( disk./.usagePercent‚78.0440240488972 disk.drbd1260.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚171851.5673828125ßÖN)‚î‚(:÷ …üçÔÏ´ñz disk.drbd1260.out‚ 0.99609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3263.91796875 disk.sda3.out‚ 216.3671875) disk./.usagePercent‚78.04243219404928) disk.nvme0c0n1.out‚172246.43994140625 disk.dm-0.in‚ 18608.2265625 disk.sda.in‚0 disk.sda.out‚ 216.3671875! disk.nvme0n1.in‚ 18608.2265625' proc.memory.availableMB‚ 711.8203125# disk.nvme0c0n1.in‚ 18608.2265625 disk.drbd1260.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0$ disk.dm-0.out‚172246.43994140625 disk.dm-1.in‚0 disk.dm-1.out‚ 216.3671875' disk.nvme0n1.out‚172246.43994140625$ memory_percent‚95.65539360046387* proc.memory.percent‚19.921374320983887 cpu‚25.159072710668532 network.sent‚ 1.0685794e+07 network.recv‚ 2.24949e+06% disk./.usageGB‚1162.8886375427246X¸ó>‘ï‚(:† …üçԿ΃| gpu.0.gpu‚44.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚207.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t¨ìcSð‚(:ø ”üçÔ‘ãx network.recv‚ 2.252627e+06) disk./.usagePercent‚78.01329469653973% disk./.usageGB‚1162.4544677734375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 18608.2265625) disk.nvme0c0n1.out‚172903.35400390625' disk.nvme0n1.out‚172903.35400390625 disk.sda1.out‚0 disk.sda.out‚ 216.7421875 disk.sda3.in‚0 disk.sda3.out‚ 216.7421875 cpu‚25.158896789509765 network.sent‚ 1.0690683e+07 disk.sda2.in‚0 disk.dm-0.in‚ 18608.2265625$ disk.dm-0.out‚172903.35400390625 disk.dm-1.in‚0 disk.dm-1.out‚ 216.7421875 disk.drbd1GqŸì0260.in‚0# memory_percent‚95.6568956375122! disk.nvme0n1.in‚ 18608.2265625 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd1260.out‚ 0.99609375( proc.memory.availableMB‚ 711.57421875# proc.memory.rssMB‚ 3263.91796875* proc.memory.percent‚19.921374320983887 proc.cpu.threads‚17@Ç/‘œñ‚(:‘ ”üçÔòˆy gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`–nò‚(:ã £üçÔË»Ãx disk.dm-1.in‚0* proc.memory.percent‚19.921374320983887 cpu‚25.15867383839533 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚173116.923828125 disk.sda.in‚0 disk.sda.out‚ 217.296875 disk.drbd1260.out‚1( proc.memory.availableMB‚ 711.54296875 network.recv‚ 2.257815e+06% disk.nvme0n1.out‚173116.923828125 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 18608.2265625 disk.dm-0.in‚ 18608.2265625 disk.dm-1.out‚ 217.296875 disk.drbd1260.in‚0 network.sent‚ 1.0697181e+07# disk./.usageGB‚1162.47847366333! disk.nvme0n1.in‚ 18608.2265625' disk.nvme0c0n1.out‚173116.923828125 disk.sda3.in‚0 disk.sda3.out‚ 217.296875$ memory_percent‚95.65708637237549# proc.memory.rssMB‚ 3263.91796875( disk./.usagePercent‚78.0149057519527\²Èý›ó‚(: £üçÔÓλx gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.323' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.464599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xútâvô‚(:ë ²üçÔ¯Íày% disk./.usageGB‚1162.4784545898438 disk.sda.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚95.65706253051758* proc.memory.percent‚19.921374320983887 network.sent‚ 1.0702036e+07) disk./.usagePercent‚78.01490447191503 disk.dm-0.in‚ 18608.2265625" disk.dm-0.out‚173367.427734375 disk.sda1.out‚0& proc.memory.availableMB‚ 711.546875 proc.cpu.threads‚17 network.recv‚ 2.261077e+06# disk.nvme0c0n1.in‚ 18608.2265625' disk.nvme0c0n1.out‚173367.427734375 disk.dm-1.in‚0 disk.sda.out‚ 217.65234375 disk.drbd1260.in‚0 disk.drbd1260.out‚1% disk.nvme0n1.out‚173367.427734375 disk.dm-1.out‚ 217.65234375 disk.sda3.in‚0 disk.sda3.out‚ 217.65234375! disk.nvme0n1.in‚ 18608.2265625 disk.sda2.in‚0# proc.memory.rssMB‚ 3263.91796875 cpu‚25.158509832880892r²7œõ‚(:‘ ²üçÔÖõ°z gpu.0.gpu‚44.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚189.976' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.995200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚581487566913024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚622364469052308 $ _timestamp‚1.788476978552983e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627812246969856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52914081 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9628052537344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36083969 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚622360834961656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88998045 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88998759 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚637436688846336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98949825 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚637436688846336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627812273361408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612753272542736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83626335 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83626335 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚637440325898752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568237510770176 _step‚322 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719IiFÇ(û€P ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9624441876480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98976165 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9611171081604 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98976165 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83626341 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9628052537344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568237561973248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9607562418920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚637436688846336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627812273361408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612753272542736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83626335 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83626335 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚637440325898752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568237510770176 _step‚322 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83626341 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9624441876480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚3949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182576160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚612753297970704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚581487566913024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚622364469052308 $ _timestamp‚1.788476978552983e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627812246969856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52914081 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9628052537344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36083969 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚622360834961656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88998045 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88998759 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚637436688846336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98949825 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182647523 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13253240976896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚581630042136064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9613577385098 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98974392 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622517204596938 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182621151 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83646765 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚581630169458176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚637596513952768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚637592876887040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚612903627211840 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83646759 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99000758 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 trainer/global_step‚3949 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9626852358144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36093345 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36092662 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 % _timestamp‚1.7884769795909066e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytesQ(gö….small_pool.current‚76119040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚622513570506078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627966050920448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612903601783872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚637596513952768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89019543 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627966024528896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99000758 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9626852358144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52926886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚323 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182621151 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89020231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 _runtime‚7186.347496682IXÆ@'þ‹N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36092662 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 % _timestamp‚1.7884769795909066e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚622513570506078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627966050920448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612903601783872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚637596513952768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89019543 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627966024528896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99000758 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9626852358144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52926886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚323 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182621151 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89020231 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 _runtime‚7186.347496682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9609968722206 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627966050920448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚568376801159168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182647523 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13253317095936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9630463032320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83646765 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182647523 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13253240976896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚581630042136064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9613577385098 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98974392 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚622517204596938 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182621151 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83646765 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚581630169458176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚637596513952768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚637592876887040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚612903627211840 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83646759 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99000758 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 trainer/global_step‚3949 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9626852358144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36093345 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187Ãb\¢2ÿ- _wandbruntime‚7186  _runtime‚7186¹©ÈY€Ó /loss/FRt4/detection_loss‚2.4161578700655983 /loss/FRt1/detection_loss‚1.6298708098082508 /loss/FRt3/assignment_loss‚4.839197528362274 0loss/FRt4/assignment_loss‚3.2485699894882383 2loss/SRqqt3/assignment_loss‚1.8698149599755804 1loss/SRqqt4/assignment_loss‚2.280511498120096 0loss/FRt1/assignment_loss‚2.2279627782957894 0loss/FBt3/assignment_loss‚0.5313373199896887 0loss/SRqqt2/detection_loss‚2.943583378791809 0loss/SRqqt4/detection_loss‚2.157551148202684 /loss/FBt3/detection_loss‚2.1957165400187173 /loss/FBt4/detection_loss‚2.4610195093684726 _step‚324 .loss/FRt3/detection_loss‚3.348505985736847 % _timestamp‚1.7884769795936654e+09 _runtime‚7186.349100243 1loss/SRqqt1/detection_loss‚1.1443835725195897 0loss/SRqqt3/detection_loss‚2.599381680289904 /loss/FBt2/detection_loss‚0.7743814752210636 epoch‚3 trainer/global_step‚3949 2loss/SRqqt1/assignment_loss‚1.2467131514229581 2loss/SRqqt2/assignment_loss‚1.9208987486362457 /loss/FBt1/assignment_loss‚0.468136224197201 .loss/FBt1/detection_loss‚2.954805600643158 1loss/FBt2/assignment_loss‚0.26471375101037165 /loss/FRt2/detection_loss‚1.9165417372918965 1loss/FBt4/assignment_loss‚0.38041714017164296 0loss/FRt2/assignment_loss‚2.5098132689793906 %loss/total_loss‚48.32998566660746c–„^Ø /loss/FRt1/detection_loss‚1.6298708098082508 /loss/FRt3/assignment_loss‚4.839197528362274 0loss/FRt4/assignment_loss‚3.2485699894882383 2loss/SRqqt3/assignment_loss‚1.8698149599755804 1loss/SRqqt4/assignment_loss‚2.280511498120096 0loss/FRt1/assignment_loss‚2.2279627782957894 0loss/FBt3/assignment_loss‚0.5313373199896887 0loss/SRqqt2/detection_loss‚2.943583378791809 0loss/SRqqt4/detection_loss‚2.157551148202684 /loss/FBt3/detection_loss‚2.1957165400187173 /loss/FBt4/detection_loss‚2.4610195093684726 _step‚324 .loss/FRt3/detection_loss‚3.348505985736847 % _timestamp‚1.7884769795936654e+09 _runtime‚7186.349100243 1loss/SRqqt1/detection_loss‚1.1443835725195897 0loss/SRqqt3/detection_loss‚2.599381680289904 /loss/FBt2/detection_loss‚0.7743814752210636 epoch‚3 trainer/global_step‚3949 2loss/SRqqt1/assignment_loss‚1.2467131514229581 2loss/SRqqt2/assignment_loss‚1.9208987486362457 /loss/FBt1/assignment_loss‚0.468136224197201 .loss/FBt1/detection_loss‚2.954805600643158 1loss/FBt2/assignment_loss‚0.26471375101037165 /loss/FRt2/detection_loss‚1.9165417372918965 1loss/FBt4/assignment_loss‚0.38041714017164296 0loss/FRt2/assignment_loss‚2.5098132689793906 %loss/total_loss‚48.32998566660746 /loss/FRt4/detection_loss‚2.4161578700655983ĈŒE{‚‚(:ð ÁüçÔâóîy* proc.memory.percent‚19.921374320983887% disk./.usageGB‚1162.4784393310547 disk.dm-1.in‚0 disk.sda.out‚ 218.06640625 disk.drbd1260.out‚ 1.00390625# proc.memory.rssMB‚ 3263.91796875 proc.cpu.threads‚17 network.recv‚ 2.2658e+06 disk.dm-1.out‚ 218.06640625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 18758.1796875 network.sent‚ 1.0708295e+07 disk.sda.in‚0 disk.sda3.out‚ 218.06640625! disk.nvme0n1.in‚ 18758.1796875 disk.sda1.in‚0$ memory_percent‚95.65582275390625" proc.memory.availableMB‚711.75 cpu‚25.158697301099302( disk./.usagePercent‚78.0149034478849& disk.nvme0n1.out‚173619.1767578125( disk.nvme0c0n1.out‚173619.1767578125 disk.dm-0.in‚ 18758.1796875# disk.dm-0.out‚173619.1767578125O 0<‘ƒ‚(:† ÁüçÔãûy gpu.0.gpu‚13.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>²o’„‚(:‡ ÐüçÔ¹‚Ùw gpu.0.gpu‚87.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚252.272' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.4544 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ûU „…‚(:ù ÐüçÔ°Ðôy disk.sda.in‚0 disk.sda3.out‚ 218.44140625! disk.nvme0n1.in‚ 18758.1796875& disk.nvme0n1.out‚174060.8955078125 network.recv‚ 2.271147e+06# disk.nvme0c0n1.in‚ 18758.1796875# disk.dm-0.out‚174060.8955078125 disk.sda3.in‚0 disk.drbd1260.out‚ 1.00390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 218.44140625 disk.sda1.in‚0$ memory_percent‚95.65579891204834( proc.memory.availableMB‚ 711.75390625# proc.memory.rssMB‚ 3263.91796875* proc.memory.percent‚19.921374320983887% disk./.usageGB‚1162.4545936584473 disk.dm-1.in‚0 disk.sda.out‚ 218.44140625 disk.drbd1260.in‚0 cpu‚25.158918235670356 proc.cpu.threads‚17 network.sent‚ 1.0817511e+07) disk./.usagePercent‚78.01330314478835( disk.nvme0c0n1.out‚174060.8955078125 disk.dm-0.in‚ 18758.1796875‚$ú†‚(:ö ßüçÔçëáy& disk.nvme0n1.out‚174503.8720703125 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 18758.1796875 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚95.65727710723877! disk.nvme0n1.in‚ 18758.1796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 18758.1796875# disk.dm-0.out‚174503.8720703125 disk.sda.out‚ 218.87109375 disk.drbd1260.out‚ 1.0078125* proc.memory.percent‚19.921374320983887 network.sent‚ 1.0822312e+07 network.recv‚ 2.276397e+06) disk./.usagePercent‚78.01487682310133 disk.sda3.out‚ 218.87109375# proc.memory.rssMB‚ 3263.91796875 cpu‚25.15910582244798( disk.nvme0c0n1.out‚174503.8720703125 disk.dm-1.out‚ 218.87109375 disk.sda.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 711.51171875 proc.cpu.threads‚17$ disk./.usageGB‚1162.478042602539wß©G‡‚(:… ßüçÔ¹ÈÚz gpu.0.gpu‚15.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚136.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.293 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ä fˆ‚(:’ îüçÔ ›„w gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚272.585' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.516999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃxFW€‰‚(:õ îüçÔãüÛy% disk./.usageGB‚1161.8265380859375 disk.sda.out‚ 219.23828125 disk.sda3.in‚0 disk.drbd1260.out‚ 1.0078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 18758.1796875# disk.dm-0.out‚175020.0947265625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 219.23828125 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 711.515625# disk.nvme0c0n1.in‚ 18758.1796875 disk.dm-1.out‚ 219.23828125# proc.memory.rssMB‚ 3263.91796875* proc.memory.percent‚19.921374320983887 cpu‚25.158985313772536 proc.cpu.threads‚17 network.recv‚ 2.279103e+06( disk.nvme0c0n1.out‚175020.0947265625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 18758.1796875& disk.nvme0n1.out‚175020.0947265625$ memory_percent‚95.65725326538086 network.sent‚ 1.0827093e+07( disk./.usagePercent‚77.9711538083435€ú÷2Š- _wandbruntime‚7259  _runtime‚7259Š ÒÔÄ‹¾ & lr-AdamW/pg1‚0.0005517029616669322 & lr-AdamW/pg2‚0.0005517029616669322 trainer/global_step‚3999 # _timestamp‚1.78847705280699e+09 _runtime‚ 7259.56261186 _step‚325é»4Ɍà & lr-AdamW/pg1‚0.0005517029616669322 & lr-AdamW/pg2‚0.0005517029616669322 trainer/global_step‚3999 # _timestamp‚1.78847705280699e+09 _runtime‚ 7259.56261186 _step‚325ÅàAí2- _wandbruntime‚7259  _runtime‚7259pÈé(Ž€P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚645249742347776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚645246105295360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588618214704128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13416059246080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚635501124921856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36523932 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚620269760027504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184855408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9727891174428 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 _step‚326 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9748591034368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9744980373504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84647541 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90078959 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90078258 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100207867 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645249742347776 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53554331 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645246105295360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184829062 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9748591034368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100207867 % _timestamp‚1.7884770528082428e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚7259.564167391 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13416133281280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100181527 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚629997625773964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84647541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36524628 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575202030219776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588618089465856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84647541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36524628 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575202030219776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588618089465856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84647541 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90078959 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90078258 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100207867 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚645249742347776 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53554331 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚645246105295360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184829062 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9748591034368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100207867 % _timestamp‚1.7884770528082428e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚7259.564167391 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13416133281280Æ yU‚‚(:÷ ýüçÔ½þá{ disk.sda1.in‚0$ disk.dm-0.out‚175259.79150390625 disk.dm-1.in‚0$ memory_percent‚95.65892219543457# proc.memory.rssMB‚ 3263.91796875) disk./.usagePercent‚77.97276076763593 disk.sda3.in‚0 disk.drbd1260.out‚ 1.01171875 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 18773.6484375 disk.dm-0.in‚ 18773.6484375* proc.memory.percent‚19.921374320983887 cpu‚25.15878298609202 network.sent‚ 1.0832964e+07 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚175259.79150390625 disk.dm-1.out‚ 219.7265625' proc.memory.availableMB‚ 711.2421875 proc.cpu.threads‚17% disk./.usageGB‚1161.8504829406738 disk.sda.in‚0! disk.nvme0n1.in‚ 18773.6484375' disk.nvme0n1.out‚175259.79150390625 disk.sda2.in‚0 disk.sda1.out‚0 network.recv‚ 2.283976e+06 disk.sda.out‚ 219.7265625 disk.sda3.out‚ 219.7265625æà(•’‘‚(:‡ ýüçÔüšŸ| gpu.0.gpu‚90.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚119.176' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8352 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µñ2’- _wandbruntime‚7260  _runtime‚7260´jÀ,'“ŠN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 _step‚327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184900425 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚620420063840672 trainer/global_step‚3999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36533415 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184874053 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575341320608768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚630150361318386 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53567136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚620420089268640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588760702657536 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84667959 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9751001529344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635654902480896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9751001529344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90099863 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84667965 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚645405930401792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13419456070656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9747390855168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575341371811840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13419382048768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 _runtime‚7260.668258169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9733906140606 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 % _timestamp‚1.7884770539103072e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184874053 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100206094 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100232460 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728îY´h'”N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100206094 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100232460 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 _step‚327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184900425 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚620420063840672 trainer/global_step‚3999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36533415 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184874053 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575341320608768 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚630150361318386 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53567136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚620420089268640 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588760702657536 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84667959 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9751001529344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚635654902480896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9751001529344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90099863 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84667965 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚645405930401792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13419456070656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9747390855168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575341371811840 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13419382048768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 _runtime‚7260.668258169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9733906140606 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 % _timestamp‚1.7884770539103072e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184874053 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Çót›2•- _wandbruntime‚7260  _runtime‚7260ÞàiY–Ó _step‚328 /loss/FRt2/assignment_loss‚3.003505293969755 1loss/FBt3/assignment_loss‚0.44760445111185415 /loss/FRt4/detection_loss‚2.5871569173676625 1loss/SRqqt2/detection_loss‚2.6092693458000817 &loss/total_loss‚47.437419088815695 _runtime‚7260.669692943 1loss/FBt2/assignment_loss‚0.22478924720345508 /loss/FBt1/detection_loss‚1.8500481375626154 2loss/SRqqt2/assignment_loss‚2.4820279648734465 0loss/SRqqt4/detection_loss‚2.198769516066501 /loss/FBt2/detection_loss‚0.7917527781048818 /loss/FRt3/assignment_loss‚4.512931880496797 2loss/SRqqt3/assignment_loss‚1.8120598842700322 1loss/FBt1/assignment_loss‚0.14006994336524972 /loss/FRt2/detection_loss‚2.0728957606134593 .loss/FBt4/detection_loss‚2.725275116808274 trainer/global_step‚3999 /loss/FRt1/assignment_loss‚2.053457104954226 2loss/SRqqt1/assignment_loss‚1.1500324649054423 0loss/FBt4/assignment_loss‚0.3319870568537975 .loss/FRt1/detection_loss‚1.709214620083787 1loss/SRqqt1/detection_loss‚1.1722262620127626 % _timestamp‚1.7884770539142098e+09 .loss/FBt3/detection_loss‚2.333972887860404 1loss/SRqqt4/assignment_loss‚1.807336902932117 1loss/SRqqt3/detection_loss‚2.6848827550808587 epoch‚3 /loss/FRt4/assignment_loss‚3.504437526067098 .loss/FRt3/detection_loss‚3.231715270451137@ÔÜ^—Ø 2loss/SRqqt1/assignment_loss‚1.1500324649054423 0loss/FBt4/assignment_loss‚0.3319870568537975 .loss/FRt1/detection_loss‚1.709214620083787 1loss/SRqqt1/detection_loss‚1.1722262620127626 % _timestamp‚1.7884770539142098e+09 .loss/FBt3/detection_loss‚2.333972887860404 1loss/SRqqt4/assignment_loss‚1.807336902932117 1loss/SRqqt3/detection_loss‚2.6848827550808587 epoch‚3 /loss/FRt4/assignment_loss‚3.504437526067098 .loss/FRt3/detection_loss‚3.231715270451137 _step‚328 /loss/FRt2/assignment_loss‚3.003505293969755 1loss/FBt3/assignment_loss‚0.44760445111185415 /loss/FRt4/detection_loss‚2.5871569173676625 1loss/SRqqt2/detection_loss‚2.6092693458000817 &loss/total_loss‚47.437419088815695 _runtime‚7260.669692943 1loss/FBt2/assignment_loss‚0.22478924720345508 /loss/FBt1/detection_loss‚1.8500481375626154 2loss/SRqqt2/assignment_loss‚2.4820279648734465 0loss/SRqqt4/detection_loss‚2.198769516066501 /loss/FBt2/detection_loss‚0.7917527781048818 /loss/FRt3/assignment_loss‚4.512931880496797 2loss/SRqqt3/assignment_loss‚1.8120598842700322 1loss/FBt1/assignment_loss‚0.14006994336524972 /loss/FRt2/detection_loss‚2.0728957606134593 .loss/FBt4/detection_loss‚2.725275116808274 trainer/global_step‚3999 /loss/FRt1/assignment_loss‚2.053457104954226Ȭ%…Þ˜‚(:’ ŒýçÔ„ú•w gpu.0.gpu‚75.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚278.849' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.769800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÖFEl™‚(:þ ŒýçÔ¿©‚y$ memory_percent‚95.65770626068115 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚ÒOí¼18894.60546875( proc.memory.availableMB‚ 711.44140625 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 220.13671875 disk.sda3.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚175452.31884765625 disk.dm-1.out‚ 220.13671875* proc.memory.percent‚19.921374320983887 cpu‚25.158682422793778 network.sent‚ 1.0932058e+07( disk./.usagePercent‚77.9727635837188% disk./.usageGB‚1161.8505249023438 disk.sda3.out‚ 220.13671875 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚18894.60546875# proc.memory.rssMB‚ 3263.91796875 network.recv‚ 2.295347e+06 disk.drbd1260.out‚ 1.01171875" disk.nvme0n1.in‚18894.60546875' disk.nvme0n1.out‚175452.31884765625 disk.sda1.out‚0$ disk.dm-0.out‚175452.31884765625 disk.dm-1.in‚0ÓvÛ‘š‚(:† ›ýçÔÔ†”| gpu.0.gpu‚14.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚206.242' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.2484 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÐBit›‚(:é ›ýçÔ¡Ô}" disk.nvme0n1.in‚18894.60546875 disk.sda2.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚18894.60546875' disk.nvme0n1.out‚175717.41259765625) disk.nvme0c0n1.out‚175717.41259765625 disk.dm-0.in‚18894.60546875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚220.5 disk.sda3.in‚0( proc.memory.availableMB‚ 711.35546875 network.sent‚ 1.1036947e+07 network.recv‚ 2.309849e+06 disk.dm-1.out‚220.5 disk.sda3.out‚220.5$ memory_percent‚95.65823078155518) proc.memory.percent‚19.92175579071045 cpu‚25.158598205152312 disk.sda2.in‚0 disk.sda1.in‚0$ disk.dm-0.out‚175717.41259765625# proc.memory.rssMB‚ 3263.98046875 proc.cpu.threads‚17) disk./.usagePercent‚77.97118529727022% disk./.usageGB‚1161.8270072937012 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.01953125úxeZ‚œ‚(:÷ ªýçÔ´ª¦y" disk.dm-0.out‚175981.244140625 network.sent‚ 1.1040012e+07 network.recv‚ 2.31343e+06 disk.sda2.in‚0$ disk.nvme0c0n1.in‚18894.60546875 disk.dm-0.in‚18894.60546875 disk.sda3.out‚ 220.94921875 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚175981.244140625 disk.dm-1.out‚ 220.94921875 disk.sda.in‚0) proc.memory.percent‚19.92175579071045 cpu‚25.158983064532453) disk./.usagePercent‚77.97118401723255% disk./.usageGB‚1161.8269882202148% disk.nvme0n1.out‚175981.244140625 disk.dm-1.in‚0 disk.sda.out‚ 220.94921875$ memory_percent‚95.65675258636475# proc.memory.rssMB‚ 3263.98046875 proc.cpu.threads‚17" disk.nvme0n1.in‚18894.60546875 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 1.01953125( proc.memory.availableMB‚ 711.59765625fs‚(:„ ªýçÔÜ­’z gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚178.875' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=nk¨…ž‚(:ú ¹ýçÔ‘‘Âx disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-0.in‚18894.60546875$ disk.dm-0.out‚176343.56201171875# proc.memory.rssMB‚ 3263.98046875 network.sent‚ 1.1046718e+07' disk.nvme0n1.out‚176343.56201171875 disk.sda2.out‚0$ memory_percent‚95.65668106079102) proc.memory.percent‚19.92175579071045 disk.dm-1.out‚ 221.35546875 disk.sda3.out‚ 221.35546875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.0234375" disk.nvme0n1.in‚18894.60546875 disk.sda2.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 2.318872e+06 disk.sda.in‚0 disk.sda.out‚ 221.35546875$ disk.nvme0c0n1.in‚18894.60546875) disk.nvme0c0n1.out‚176343.56201171875& proc.memory.availableMB‚ 711.609375 cpu‚25.15893671327944) disk./.usagePercent‚77.96448353203887% disk./.usageGB‚1161.7271461486816båÉ ’Ÿ‚(:‡ ¹ýçÔæü½x gpu.0.gpu‚80.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚278.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±žI2 - _wandbruntime‚7329  _runtime‚7329ùû`ˆÇ¡Á trainer/global_step‚4049 % _timestamp‚1.7884771232213998e+09 _runtime‚7329.977452486 _step‚329 & lr-AdamW/pg1‚0.0005482862140966141 & lr-AdamW/pg2‚0.0005482862140966141SzµºÌ¢Æ % _timestamp‚1.7884771232213998e+09 _runtime‚7329.977452486 _step‚329 & lr-AdamW/pg1‚0.0005482862140966141 & lr-AdamW/pg2‚0.0005482862140966141 trainer/global_step‚4049ÉÅb2£- _wandbruntime‚7329  _runtime‚73290’·M(¤€P eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚582166600872448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9848219929936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚726 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54194581 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚643190002873856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36964969 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9851828592620 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚582166549669376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627786196656336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9869129531392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36965690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚643190029265408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9865518870528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101413229 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101439569 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85668735 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595749031889920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13582431017472 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85668741 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101413229 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101439569 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13582356982272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7329.978905645 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9869129531392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54194576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚721 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85668735 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627786222084304 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187081964 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85668741 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚653059158796800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚637634416586272 trainer/global_step‚4049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9865518870528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚653055521744384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚653055521744384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚643190029265408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643190002873856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884771232226155e+09ˆWÀ/ (¥…P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚653055521744384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚643190029265408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643190002873856 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884771232226155e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚582166600872448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9848219929936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚726 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54194581 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚643190002873856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36964969 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9851828592620 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚582166549669376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627786196656336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9869129531392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36965690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚643190029265408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9865518870528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101413229 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101439569 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85668735 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595749031889920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13582431017472 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85668741 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101413229 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101439569 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13582356982272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7329.978905645 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9869129531392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54194576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚721 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85668735 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627786222084304 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187081964 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85668741 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚653059158796800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚637634416586272 trainer/global_step‚4049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9865518870528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚653055521744384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904ÊmÉ2¦- _wandbruntime‚7331  _runtime‚7331!%°§ŠN F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚653215346850816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9871540026368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85689159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36974404 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101437796 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9854234896114 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187126955 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85689159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582305891261440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85689165 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9867929352192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91181790 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 _runtime‚7331.159020241 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101464162 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595891526336000 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9867929352192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚653211709785088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101464162 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54207386 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚637787152130694 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 % _timestamp‚1.7884771244021823e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91181087 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9850626233222 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54207381 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627936525897472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643343806824448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187153327 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚582305840058368 _step‚331 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚637790786221554 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13585762396672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚643343806824448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9871540026368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187153327 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187126955 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚582305840058368 _step‚331 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚637790786221554 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13585762396672 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚643343806824448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9871540026368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187153327 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187126955 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚653215346850816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9871540026368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85689159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36974404 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101437796 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9854234896114 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187126955 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85689159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582305891261440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85689165 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9867929352192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91181790 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 _runtime‚7331.159020241 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101464162 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595891526336000 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4049 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9867929352192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚653211709785088 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101464162 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54207386 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚637787152130694 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 % _timestamp‚1.7884771244021823e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91181087 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9850626233222 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54207381 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627936525897472Ëã»àr2©- _wandbruntime‚7331  _runtime‚7331Xhþ)SªÍ /loss/FRt3/assignment_loss‚3.848800474597562 /loss/FRt1/detection_loss‚1.5006412892814427 0loss/SRqqt4/detection_loss‚2.428329737484455 .loss/FBt4/detection_loss‚2.726705409013308 epoch‚3 trainer/global_step‚4049 /loss/FRt1/assignment_loss‚2.179345365791094 /loss/FBt2/detection_loss‚0.8343570278315194 2loss/SRqqt2/assignment_loss‚1.7715233626465003 /loss/FRt3/detection_loss‚3.0724367557033414 # _timestamp‚1.78847712440474e+09 1loss/SRqqt1/assignment_loss‚1.303319271272201 2loss/SRqqt3/assignment_loss‚1.9164537474130974 0loss/FBt2/assignment_loss‚0.2433573977442091 0loss/FBt4/assignment_loss‚0.2611478136932419 /loss/FBt3/detection_loss‚3.0696902091686544 _step‚332 0loss/FRt2/assignment_loss‚2.6813514590263368 /loss/FRt4/assignment_loss‚4.547249910831451 2loss/SRqqt4/assignment_loss‚1.7402611257508398 1loss/FBt3/assignment_loss‚0.17200396470434498 .loss/FRt4/detection_loss‚3.118606481552124 0loss/SRqqt2/detection_loss‚2.694310468931993 0loss/FBt1/assignment_loss‚0.6214368183282204 .loss/FBt1/detection_loss‚4.105795323848724 %loss/total_loss‚50.98861583374515 .loss/FRt2/detection_loss‚2.779167738556862 1loss/SRqqt1/detection_loss‚1.0348408572742196 0loss/SRqqt3/detection_loss‚2.337483823299408 _runtime‚7331.160221165[º7X«Ò 0loss/FBt1/assignment_loss‚0.6214368183282204 .loss/FBt1/detection_loss‚4.105795323848724 %loss/total_loss‚50.98861583374515 .loss/FRt2/detection_loss‚2.779167738556862 1loss/SRqqt1/detection_loss‚1.0348408572742196 0loss/SRqqt3/detection_loss‚2.337483823299408 _runtime‚7331.160221165 /loss/FRt3/assignment_loss‚3.848800474597562 /loss/FRt1/detection_loss‚1.5006412892814427 0loss/SRqqt4/detection_loss‚2.428329737484455 .loss/FBt4/detection_loss‚2.726705409013308 epoch‚3 trainer/global_step‚4049 /loss/FRt1/assignment_loss‚2.179345365791094 /loss/FBt2/detection_loss‚0.8343570278315194 2loss/SRqqt2/assignment_loss‚1.7715233626465003 /loss/FRt3/detection_loss‚3.0724367557033414 # _timestamp‚1.78847712440474e+09 1loss/SRqqt1/assignment_loss‚1.303319271272201 2loss/SRqqt3/assignment_loss‚1.9164537474130974 0loss/FBt2/assignment_loss‚0.2433573977442091 0loss/FBt4/assignment_loss‚0.2611478136932419 /loss/FBt3/detection_loss‚3.0696902091686544 _step‚332 0loss/FRt2/assignment_loss‚2.6813514590263368 /loss/FRt4/assignment_loss‚4.547249910831451 2loss/SRqqt4/assignment_loss‚1.7402611257508398 1loss/FBt3/assignment_loss‚0.17200396470434498 .loss/FRt4/detection_loss‚3.118606481552124 0loss/SRqqt2/detection_loss‚2.694310468931993Ìëè~rt¬‚(:é ÈýçÔǺÚx# memory_percent‚95.6568956375122) proc.memory.percent‚19.92175579071045 proc.cpu.threads‚17) disk./.usagePercent‚77.90714373254234 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-0.out‚176809.8359375 disk.dm-1.out‚ 221.6953125( proc.memory.availableMB‚ 711.57421875 cpu‚25.15878267857071 network.recv‚ 2.322228e+06% disk./.usageGB‚1160.8727416992188 disk.sda3.out‚ 221.6953125# disk.nvme0n1.out‚176809.8359375 disk.sda1.in‚0 disk.drbd1260.out‚ 1.0234375! disk.nvme0n1.in‚ 19003.8046875 disk.sda2.out‚0% disk.nvme0c0n1.out‚176809.8359375 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 221.6953125# proc.memory.rssMB‚ 3263.98046875 network.sent‚ 1.1051561e+07 disk.drbd1260.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 19003.8046875 disk.dm-0.in‚ 19003.8046875>Zñ’­‚(:‡ ÈýçÔŸÆðy gpu.0.gpu‚62.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚116.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)+´ {®‚(:ð ×ýçÔò½x disk.drbd1260.out‚ 1.02734375 disk.sda2.in‚0) disk.nvme0c0n1.out‚177318.27880859375 disk.sda.in‚0 disk.sda.out‚ 222.078125$ proc.memory.availableMB‚711.5625 cpu‚25.158968085014166 proc.cpu.threads‚17) disk./.usagePercent‚77.90707461050809$ disk./.usageGB‚1160.871711730957 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 19003.8046875 disk.dm-0.in‚ 19003.8046875 disk.dm-1.out‚ 222.078125$ memory_percent‚95.65696716308594 network.sent‚ 1.1170242e+07! disk.nvme0n1.in‚ 19003.8046875 disk.dm-1.in‚0# proc.memory.rssMB‚ 3263.98046875) proc.memory.percent‚19.92175579071045 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 2.348332e+06 disk.sda3.out‚ 222.078125 disk.drbd1260.in‚0' disk.nvme0n1.out‚177318.27880859375 disk.sda2.out‚0$ disk.dm-0.out‚177318.27880859375'Ä;‘¯‚(:† ×ýçÔ°šÓx gpu.0.gpu‚16.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚212.911' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.5822 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÅø ~°‚(:ó æýçÔà‡‚x network.recv‚ 2.351694e+06) disk.nvme0c0n1.out‚177442.48193359375 disk.dm-0.in‚ 19003.8046875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 222.5078125 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.98046875% disk./.usageGB‚1160.8764381408691 disk.dm-1.in‚0 disk.dm-1.out‚ 222.5078125! disk.nvme0n1.in‚ 19003.8046875' disk.nvme0n1.out‚177442.48193359375$ proc.memory.availableMB‚711.5625 cpu‚25.15895330881706 proc.cpu.threads‚17) disk./.usagePercent‚77.90739180384301# disk.nvme0c0n1.in‚ 19003.8046875$ disk.dm-0.out‚177442.48193359375 disk.sda.out‚ 222.5078125 disk.drbd1260.out‚ 1.02734375 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚95.65696716308594 network.sent‚ 1.1175074e+07 disk.drbd1260.in‚0 disk.sda2.out‚0) proc.memory.percent‚19.92175579071045£ëKd‘±‚(:† æýçÔ›‰°x gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚112.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5696 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|¹]ù’²‚(:‡ õýçÔ…™¨w gpu.0.gpu‚58.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚231.926' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.3852 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06‰5‹³‚(:ô õýçÔÜÇ”x network.sent‚ 1.1191264e+07$ disk./.usageGB‚1160.852970123291$ disk.dm-0.out‚177538.41064453125 disk.dm-1.out‚ 223.0234375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.out‚0) disk./.usagePercent‚77.90581684549237# disk.nvme0c0n1.in‚ 19003.8046875! disk.nvme0n1.in‚ 19003.8046875# memory_percent‚95.6662654876709 proc.cpu.threads‚17 network.recv‚ 2.376345e+06) disk.nvme0c0n1.out‚177538.41064453125 disk.dm-0.in‚ 19003.8046875 disk.drbd1260.out‚ 1.02734375' disk.nvme0n1.out‚177538.41064453125' proc.memory.availableMB‚ 710.0390625 cpu‚25.15910140794511 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 223.0234375 disk.sda3.out‚ 223.0234375 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3263.98046875) proc.memory.percent‚19.92175579071045K¼Md€´‚(:õ „þçÔÖã¬x cpu‚25.15918207993217 proc.cpu.threads‚17' disk.nvme0n1.out‚178004.15673828125 network.sent‚ 1.1196142e+07 disk.dm-1.out‚ 223.37109375 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0# disk.nvme0c0n1.in‚ 19139.6953125) disk.nvme0c0n1.out‚178004.15673828125 network.recv‚ 2.379355e+06 disk.sda2.in‚0# proc.memory.rssMB‚ 3263.98046875) proc.memory.percent‚19.92175579071045 disk.dm-1.in‚0 disk.sda.out‚ 223.37109375 disk.drbd1260.out‚1.03125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 19139.6953125$ disk.dm-0.out‚178004.15673828125# memory_percent‚95.6589937210083) disk./.usagePercent‚77.91001024890323$ disk./.usageGB‚1160.915454864502 disk.sda3.out‚ 223.37109375! disk.nvme0n1.in‚ 19139.6953125 disk.sda2.out‚0( proc.memory.availableMB‚ 711.23046875ØTÜ[’µ‚(:‡ „þçÔˆ²‰z gpu.0.gpu‚87.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.748' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ­B[|¶‚(:ñ “þçÔ±’÷x network.recv‚ 2.384427e+06( disk./.usagePercent‚77.9242017705563' disk.nvme0c0n1.out‚178325.880859375 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd1260.in‚0% disk.nvme0n1.out‚178325.880859375 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.98046875) proc.memory.percent‚19.92175579071045 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 223.82421875 disk.dm-0.in‚ 19139.6953125" disk.dm-0.out‚178325.880859375 disk.dm-1.out‚ 223.82421875 disk.drbd1260.out‚1.03125! disk.nvme0n1.in‚ 19139.6953125 disk.sda2.in‚0 network.sent‚ 1.1204526e+07% disk./.usageGB‚1161.1269187927246# disk.nvme0c0n1.in‚ 19139.6953125 disk.sda.out‚ 223.82421875$ memory_percent‚95.65961360931396( proc.memory.availableMB‚ 711.12890625 cpu‚25.159359465494298vc0«œ·‚(:‘ “þçÔÙÿ’y gpu.0.gpu‚27.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚213.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çVR2¸- _wandbruntime‚7418  _runtime‚7418YÐ|ǹÁ trainer/global_step‚4099 % _timestamp‚1.7884772115794282e+09 _runtime‚7418.335111264 _step‚333 & lr-AdamW/pg1‚0.0005448231270013043 & lr-AdamW/pg2‚0.0005448231270013043:-^ÌºÆ _step‚333 & lr-AdamW/pg1‚0.0005448231270013043 & lr-AdamW/pg2‚0.0005448231270013043 trainer/global_step‚4099 % _timestamp‚1.7884772115794282e+09 _runtime‚7418.335111264Í“!š2»- _wandbruntime‚7418  _runtime‚7418Ëypa$!¼€P R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660864938193408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37406514 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚645274841489232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102644931 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92241345 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚645271207398580 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9989668028416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚650878880825856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86689935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚650878907217408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚635302684141104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102671271 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚660868575245824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602879126227968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102671271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚650878907217408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9986057367552 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9989668028416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602879253563392 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _step‚334 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9986057367552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86689935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86689941 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚650878880825856 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13748057108992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9972157348128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189334866 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13748133241344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚635302658713136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189361212 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86689941 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚589131069118976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589131120322048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86689935 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86689941 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚650878880825856 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13748057108992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9972157348128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189334866 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13748133241344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚635302658713136 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189361212 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86689941 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚589131069118976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589131120322048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660864938193408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37406514 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚645274841489232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102644931 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92241345 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚645271207398580 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9989668028416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚650878880825856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86689935 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚650878907217408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚635302684141104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102671271 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚660868575245824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709Îß³QÕ2¾- _wandbruntime‚7419  _runtime‚7419ØzVé '¿‡N OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚645427577033862 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189379857 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189406229 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86710359 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92262883 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189379857 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54847636 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚589270359507968 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚651032684776448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37414550 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102695864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚651032658384896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102695864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9970954988730 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86710365 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92262181 " _timestamp‚1.7884772131153e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86710365 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86710359 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚651032684776448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚661024763299840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9988467849216 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚635453013382240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13751389069312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚645423942943002 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102669498 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚603021875899392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7419.871753243 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189406229 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37415247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102695864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚651032658384896 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102695864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9970954988730 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86710365 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92262181 " _timestamp‚1.7884772131153e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86710365 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86710359 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚651032684776448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚661024763299840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9988467849216 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚635453013382240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13751389069312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚645423942943002 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102669498 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚603021875899392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7419.871753243 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189406229 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37415247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189406229 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86710359 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92262883 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4099ÏË2Á- _wandbruntime‚7419  _runtime‚7419<¶g_ÂÖ /loss/FRt1/assignment_loss‚2.137363051300618 .loss/FRt2/assignment_loss‚2.59000350191043 0loss/FRt4/assignment_loss‚3.6789785986361294 2loss/SRqqt2/assignment_loss‚1.5324783793517522 epoch‚3 _runtime‚7419.873567502 /loss/FBt3/detection_loss‚2.6070597673717297 1loss/FBt3/assignment_loss‚0.37821864034661923 1loss/SRqqt2/detection_loss‚2.9675555995532443 1loss/SRqqt3/detection_loss‚2.7621288610541304 0loss/FRt3/assignment_loss‚3.7279026093690293 2loss/SRqqt3/assignment_loss‚1.7241079912561437 /loss/FRt2/detection_loss‚1.8173356392205908 1losp¯äêýs/SRqqt1/detection_loss‚0.9614273675026432 0loss/SRqqt4/detection_loss‚2.157940124764162 .loss/FBt4/detection_loss‚3.069617674900935 %loss/total_loss‚46.88074068334866 0loss/FBt4/assignment_loss‚0.5492519128422898 /loss/FBt2/detection_loss‚0.8206292296562231 1loss/SRqqt1/assignment_loss‚1.133697692274807 2loss/SRqqt4/assignment_loss‚1.8106247809003382 1loss/FBt1/assignment_loss‚0.14779764116382665 1loss/FBt2/assignment_loss‚0.27645447714370897 /loss/FRt1/detection_loss‚1.4032954228729178 /loss/FRt4/detection_loss‚3.2370319729265957 _step‚336 /loss/FRt3/detection_loss‚3.2777403230252475 /loss/FBt1/detection_loss‚2.1120994240045547 trainer/global_step‚4099 % _timestamp‚1.7884772131175096e+09¸úvÿaÃÛ 1loss/FBt2/assignment_loss‚0.27645447714370897 /loss/FRt1/detection_loss‚1.4032954228729178 /loss/FRt4/detection_loss‚3.2370319729265957 _step‚336 /loss/FRt3/detection_loss‚3.2777403230252475 /loss/FBt1/detection_loss‚2.1120994240045547 trainer/global_step‚4099 % _timestamp‚1.7884772131175096e+09 /loss/FRt1/assignment_loss‚2.137363051300618 .loss/FRt2/assignment_loss‚2.59000350191043 0loss/FRt4/assignment_loss‚3.6789785986361294 2loss/SRqqt2/assignment_loss‚1.5324783793517522 epoch‚3 _runtime‚7419.873567502 /loss/FBt3/detection_loss‚2.6070597673717297 1loss/FBt3/assignment_loss‚0.37821864034661923 1loss/SRqqt2/detection_loss‚2.9675555995532443 1loss/SRqqt3/detection_loss‚2.7621288610541304 0loss/FRt3/assignment_loss‚3.7279026093690293 2loss/SRqqt3/assignment_loss‚1.7241079912561437 /loss/FRt2/detection_loss‚1.8173356392205908 1loss/SRqqt1/detection_loss‚0.9614273675026432 0loss/SRqqt4/detection_loss‚2.157940124764162 .loss/FBt4/detection_loss‚3.069617674900935 %loss/total_loss‚46.88074068334866 0loss/FBt4/assignment_loss‚0.5492519128422898 /loss/FBt2/detection_loss‚0.8206292296562231 1loss/SRqqt1/assignment_loss‚1.133697692274807 2loss/SRqqt4/assignment_loss‚1.8106247809003382 1loss/FBt1/assignment_loss‚0.14779764116382665Ð5m8œÄ‚(:‘ ¢þçÔ¡ãÔw gpu.0.gpu‚45.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚183.152' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.630399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð$ÒtÅ‚(:é ¢þçÔ’˜}% disk.nvme0n1.out‚178479.166015625 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.98046875) proc.memory.percent‚19.92175579071045 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 19139.6953125 disk.sda.out‚ 224.21875 disk.sda3.out‚ 224.21875 disk.sda2.out‚0 cpu‚25.159477458039657( disk./.usagePercent‚77.9225935312262$ disk./.usageGB‚1161.102954864502" disk.dm-0.out‚178479.166015625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 19139.6953125$ memory_percent‚95.65930366516113 network.sent‚ 1.1207448e+07 network.recv‚ 2.387568e+06' disk.nvme0c0n1.out‚178479.166015625 disk.dm-0.in‚ 19139.6953125 disk.sda.in‚0 disk.sda2.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 711.1796875 disk.dm-1.out‚ 224.21875 disk.drbd1260.out‚ 1.03515625ÔWLUÆ‚(:ô ±þçÔûõx$ memory_percent‚95.66226005554199 proc.cpu.threads‚17 network.recv‚ 2.394388e+06% disk./.usageGB‚1161.3760604858398 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 224.71484375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.03515625' proc.memory.availableMB‚ 710.6953125) disk./.usagePercent‚77.94092187862506' disk.nvme0c0n1.out‚178906.751953125" disk.dm-0.out‚178906.751953125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 224.71484375# proc.memory.rssMB‚ 3263.98046875) proc.memory.percent‚19.92175579071045# disk.nvme0c0n1.in‚ 19139.6953125! disk.nvme0n1.in‚ 19139.6953125% disk.nvme0n1.out‚178906.751953125 disk.sda2.in‚0 cpu‚25.159654112306203 network.sent‚ 1.1318284e+07 disk.sda1.out‚0 disk.dm-0.in‚ 19139.6953125 disk.dm-1.out‚ 224.71484375 disk.sda2.out‚0´´yœÇ‚(:‘ ±þçÔß»y gpu.0.gpu‚54.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚198.223' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.644600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ç’È‚(:‡ ÀþçÔºÀ”y gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂÒ½€É‚(:õ ÀþçÔ±çy disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 225.1640625 disk.drbd1260.out‚ 1.0390625$ memory_percent‚95.66073417663574' proc.memory.availableMB‚ 710.9453125) proc.memory.percent‚19.92175579071045 disk.sda1.out‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3263.98046875 cpu‚25.15989371641736 network.sent‚ 1.1323238e+07) disk./.usagePercent‚77.93050032392121% disk./.usageGB‚1161.2207717895508" disk.nvme0n1.in‚19281.52734375& disk.nvme0n1.out‚178965.1689453125( disk.nvme0c0n1.out‚178965.1689453125 disk.dm-0.in‚19281.52734375 proc.cpu.threads‚17 network.recv‚ 2.397813e+06 disk.sda2.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚19281.52734375# disk.dm-0.out‚178965.1689453125 disk.dm-1.out‚ 225.1640625 disk.sda.out‚ 225.1640625äbƒÊ‚(:ø ÏþçÔ©Ùx network.recv‚ 2.402937e+06) disk./.usagePercent‚77.92344706034537 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚19281.52734375% disk./.usageGB‚1161.1156730651855" disk.nvme0n1.in‚19281.52734375 disk.sda1.out‚0$ disk.nvme0c0n1.in‚19281.52734375 disk.dm-1.out‚ 225.68359375 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3263.98046875) proc.memory.percent‚19.92175579071045 network.sent‚ 1.1327817e+07# disk.dm-0.out‚179288.5322265625 disk.sda.in‚0 disk.sda.out‚ 225.68359375 disk.sda3.out‚ 225.68359375$ memory_percent‚95.66068649291992& proc.memory.availableMB‚ 710.953125 proc.cpu.threads‚17& disk.nvme0n1.out‚179288.5322265625( disk.nvme0c0n1.out‚179288.5322265625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 1.0390625 cpu‚25.160079913870184‘^ëðœË‚(:‘ ÏþçԙŴy gpu.0.gpu‚46.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚209.947' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.989399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£uڀ̂(:õ ÞþçÔþˆÜx disk.sda2.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚19281.52734375 network.recv‚ 2.406201e+06 disk.sda.out‚ 225.98046875 disk.sda3.out‚ 225.98046875 disk.drbd1260.in‚0" disk.nvme0n1.in‚19281.52734375 disk.sda2.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚179610.404296875 disk.sda3.in‚0 disk.drbd1260.out‚ 1.04296875 network.sent‚ 1.1332657e+07$ disk./.usageGB‚1161.145622253418 disk.dm-0.in‚19281.52734375 disk.dm-1.in‚0' proc.memory.availableMB‚ 710.9453125# proc.memory.rssMB‚ 3263.98046875$ memory_percent‚95.66073417663574) proc.memory.percent‚19.92175579071045 cpu‚25.16022475473769 proc.cpu.threads‚17) disk./.usagePercent‚77.92545697549667" disk.dm-0.out‚179610.404296875 disk.dm-1.out‚ 225.98046875 disk.sda.in‚0% disk.nvme0n1.out‚179610.404296875R¦;œÍ‚(:‘ ÞþçÔ†åÂy gpu.0.gpu‚79.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.196' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.03920000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß»Â_΂(:… íþçÔ¸œÕw gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚138.956' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.7912 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡éfo€Ï‚(:õ íþçÔß›y disk.drbd1260.out‚ 1.04296875" disk.nvme0n1.in‚19281.52734375" disk.dm-0.out‚180058.634765625( proc.memory.availableMB‚ 710.69921875 cpu‚25.16033287006408 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 1.133884e+07 network.recv‚ 2.410711e+06% disk./.usageGB‚1161.2693901062012 disk.sda2.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚180058.634765625 disk.dm-0.in‚19281.52734375 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚19281.52734375 disk.dm-1.out‚ 226.65234375 disk.sda.in‚0 disk.sda.out‚ 226.65234375$ memory_percent‚95.66223621368408( disk./.usagePercent‚77.9337631399451 disk.sda3.out‚ 226.65234375% disk.nvme0n1.out‚180058.634765625 disk.dm-1.in‚0# proc.memory.rssMB‚ 3263.98046875) proc.memory.percent‚19.92175579071045Ò¥ 2Ð- _wandbruntime‚7511  _runtime‚7511yx?ýÇÑÁ % _timestamp‚1.7884773044963067e+09 _runtime‚7511.252135799 _step‚337 & lr-AdamW/pg1‚0.0005413144220831629 & lr-AdamW/pg2‚0.0005413144220831629 trainer/global_step‚4149·p8}ÌÒÆ & lr-AdamW/pg1‚0.0005413144220831629 & lr-AdamW/pg2‚0.0005413144220831629 trainer/global_step‚4149 % _timestamp‚1.7884773044963067e+09 _runtime‚7511.252135799 _step‚337ѦҕË2Ó- _wandbruntime‚7511  _runtime‚75114­, (Ô†P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10106595864576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87711141 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596095639771648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37846463 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93322252 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10092486103636 _runtime‚7511.253596452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚668674354642432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37847171 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10088877440952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚658567785169408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10110206525440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191614114 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87711135 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚642819120769936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103902973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚668677991694848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13913930079744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652911632301540 _step‚338 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13913853947392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610009442515968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚668674354642432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚658567785169408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10110206525440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191614114 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87711135 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚642819120769936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103902973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚668677991694848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13913930079744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚652911632301540 _step‚338 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13913853947392 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610009442515968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚668674354642432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87711141 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596095639771648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37846463 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93322252 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10092486103636 _runtime‚7511.253596452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚668674354642432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37847171 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10088877440952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚339 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚658721536336896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37855877 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103927566 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚658721562728448 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KBDeviceStatsMonitorX¼SÛ .on_train_batch_end/segment.large_pool.allocated‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103901200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10112617020416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 trainer/global_step‚4149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87731559 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596234878957568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 $ _timestamp‚1.788477306330739e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚668834179748864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596234930160640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚653060733755310 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10094892407130 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191632759 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87731565 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93343763 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653064367846170 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7513.088238687 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610152171080192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚642969475439040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10109006346240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚642969450011072 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191659131 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93343075 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13917240919552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424¥ =¹'Ø”N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610152171080192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚642969475439040 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10109006346240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚642969450011072 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191659131 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93343075 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13917240919552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚339 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚658721536336896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37855877 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103927566 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚658721562728448 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103901200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10112617020416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 trainer/global_step‚4149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87731559 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596234878957568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 $ _timestamp‚1.788477306330739e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚668834179748864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596234930160640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚653060733755310 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10094892407130 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191632759 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87731565 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93343763 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653064367846170 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7513.088238687 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Óðs‹2Ù- _wandbruntime‚7513  _runtime‚7513ö_¤ZZÚÔ % _timestamp‚1.7884773063340423e+09 2loss/SRqqt1/assignment_loss‚1.3035942111333663 2loss/SRqqt3/assignment_loss‚1.8663789062730727 .loss/FBt4/detection_loss‚2.216145120561123 epoch‚3 trainer/global_step‚4149 0loss/FBt3/assignment_loss‚0.5184170836233534 /loss/FRt3/detection_loss‚3.2133952478567758 1loss/SRqqt3/detection_loss‚2.4880047017528164 /loss/FBt1/detection_loss‚3.0185849004321628 2loss/SRqqt4/assignment_loss‚2.3107505824830796 /loss/FRt1/detection_loss‚1.8504297431805514 1loss/SRqqt2/detection_loss‚3.1816654245058698 _runtime‚7513.089690383 0loss/SRqqt1/detection_loss‚1.131982450661334 0loss/SRqqt4/detection_loss‚2.265573322110706 %loss/total_loss‚51.73185689414795 _step‚340 1loss/SRqqt2/assignment_loss‚4.473403755823771 /loss/FRt4/detection_loss‚3.2321704203883805 .loss/FBt3/detection_loss‚1.986528941988945 0loss/FRt1/assignment_loss‚1.9486438944691518 0loss/FBt1/assignment_loss‚0.3390242869176695 1loss/FBt2/assignment_loss‚0.20012338154896794 0loss/FRt2/assignment_loss‚2.8683068323135377 /loss/FRt2/detection_loss‚1.9967353639751673 /loss/FBt2/detection_loss‚0.7261615490464315 /loss/FRt3/assignment_loss‚4.559056530396144 0loss/FRt4/assignment_loss‚3.7952322214841843 0loss/FBt4/assignment_loss‚0.2415480212213879¿4Y´_ÛÙ 0loss/FBt3/assignment_loss‚0.5184170836233534 /loss/FRt3/detection_loss‚3.2133952478567758 1loss/SRqqt3/detection_loss‚2.4880047017528164 /loss/FBt1/detection_loss‚3.0185849004321628 2loss/SRqqt4/assignment_loss‚2.3107505824830796 /loss/FRt1/detection_loss‚1.8504297431805514 1loss/SRqqt2/detection_loss‚3.1816654245058698 _runtime‚7513.089690383 0loss/SRqqt1/detection_loss‚1.131982450661334 0loss/SRqqt4/detection_loss‚2.265573322110706 %loss/total_loss‚51.73185689414795 _step‚340 1loss/SRqqt2/assignment_loss‚4.473403755823771 /loss/FRt4/detection_loss‚3.2321704203883805 .loss/FBt3/detection_loss‚1.986528941988945 0loss/FRt1/assignment_loss‚1.9486438944691518 0loss/FBt1/assignment_loss‚0.3390242869176695 1loss/FBt2/assignment_loss‚0.20012338154896794 0loss/FRt2/assignment_loss‚2.8683068323135377 /loss/FRt2/detection_loss‚1.9967353639751673 /loss/FBt2/detection_loss‚0.7261615490464315 /loss/FRt3/assignment_loss‚4.559056530396144 0loss/FRt4/assignment_loss‚3.7952322214841843 0loss/FBt4/assignment_loss‚0.2415480212213879 % _timestamp‚1.7884773063340423e+09 2loss/SRqqt1/assignment_loss‚1.3035942111333663 2loss/SRqqt3/assignment_loss‚1.8663789062730727 .loss/FBt4/detection_loss‚2.216145120561123 epoch‚3 trainer/global_step‚4149Ô3—¾‘Ü‚(:† üþçÔÉÔýy gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚247.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f<pÝ‚(:å üþçÔÿ¼±z network.recv‚ 2.414024e+06% disk./.usageGB‚1161.6676406860352( disk.nvme0c0n1.out‚180507.1533203125 disk.sda2.in‚0' proc.memory.availableMB‚ 711.1953125 cpu‚25.16046857295995 proc.cpu.threads‚17 network.sent‚ 1.1343689e+07# disk.nvme0c0n1.in‚ 19369.9609375 disk.dm-1.in‚0 disk.dm-1.out‚227.375 disk.sda.out‚227.375 disk.drbd1260.in‚0& disk.nvme0n1.out‚180507.1533203125 disk.sda2.out‚0) disk./.usagePercent‚77.96049007051225 disk.dm-0.in‚ 19369.9609375# disk.dm-0.out‚180507.1533203125 disk.sda3.out‚227.375 disk.drbd1260.out‚1.046875! disk.nvme0n1.in‚ 19369.9609375 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.98046875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚95.65920829772949) proc.memory.percent‚19.92175579071045sôvGÞ‚(:… ‹ÿçÔƒïÈz gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#”‚ß‚(:÷ ‹ÿçÔš¹—{ disk.sda.in‚0 disk.sda3.in‚0 cpu‚25.16057357757999" disk.nvme0n1.in‚19403.43359375& disk.nvme0n1.out‚180780.6181640625 disk.sda1.in‚0# disk.dm-0.out‚180780.6181640625 disk.dm-1.in‚0 disk.sda.out‚ 227.67578125 disk.sda3.out‚ 227.67578125' proc.memory.availableMB‚ 710.9453125) disk./.usagePercent‚77.96048853446706 disk.sda1.out‚0$ disk.nvme0c0n1.in‚19403.43359375 disk.drbd1260.in‚0) proc.memory.percent‚19.92175579071045 proc.cpu.threads‚17 network.sent‚ 1.1454361e+07% disk./.usageGB‚1161.6676177978516 disk.dm-0.in‚19403.43359375 disk.dm-1.out‚ 227.67578125 disk.drbd1260.out‚1.046875$ memory_percent‚95.66073417663574# proc.memory.rssMB‚ 3263.98046875 network.recv‚ 2.420429e+06 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚180780.6181640625úûß«à‚(:’ šÿçÔ†º‡y gpu.0.gpu‚52.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚194.124' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.824799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l+ƒá‚(:ø šÿçÔº¤—z) disk./.usagePercent‚77.96048853446706 disk.sda1.out‚0$ disk.nvme0c0n1.in‚19403.43359375 disk.dm-1.in‚0$ memory_percent‚95.66071033477783 proc.cpu.threads‚17 network.sent‚ 1.1457349e+07 network.recv‚ 2.423787e+06 disk.sda1.in‚0# disk.dm-0.out‚180934.1845703125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 228.0546875( proc.memory.availableMB‚ 710.94921875" disk.nvme0n1.in‚19403.43359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚19403.43359375 disk.sda.out‚ 228.0546875 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3263.98046875 cpu‚25.160788734379047% disk./.usageGB‚1161.6676177978516& disk.nvme0n1.out‚180934.1845703125( disk.nvme0c0n1.out‚180934.1845703125 disk.dm-1.out‚ 228.0546875 disk.drbd1260.out‚ 1.05078125) proc.memory.percent‚19.92175579071045™K}?â‚(:ô ©ÿçÔÜ¥æx disk.sda2.out‚0# memory_percent‚95.6592321395874) disk./.usagePercent‚77.96102333420608 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.05078125( proc.memory.availableMB‚ 711.19140625 network.recv‚ 2.427748e+06 disk.sda1.in‚0# disk.dm-0.out‚181345.8564453125 disk.sda.in‚0& disk.nvme0n1.out‚181345.8564453125) proc.memory.percent‚19.92175579071045 cpu‚25.161005638248827 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚19403.43359375 disk.dm-0.in‚19403.43359375 disk.dm-1.out‚ 228.453125 disk.sda3.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚19403.43359375# proc.memory.rssMB‚ 3263.98046875 network.sent‚ 1.1462708e+07% disk./.usageGB‚1161.6755867004395( disk.nvme0c0n1.out‚181345.8564453125 disk.sda.out‚ 228.453125 disk.sda3.out‚ 228.453125 disk.sda2.in‚0ênz‘ã‚(:† ©ÿçÔ픎y gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mƒ*~ä‚(:ó ¸ÿçÔé¿Æx" disk.nvme0n1.in‚19403.43359375 disk.dm-1.out‚ 229.1796875$ memory_percent‚95.65920829772949) proc.memory.percent‚19.92175579071045 proc.cpu.threads‚17 disk.sda.out‚ 229.1796875 disk.sda3.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚181691.5869140625 cpu‚25.161181371070946 network.recv‚ 2.430912e+06$ disk./.usageGB‚1161.651123046875 disk.sda2.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚19403.43359375 disk.dm-0.in‚19403.43359375# disk.dm-0.out‚181691.5869140625# proc.memory.rssMB‚ 3263.98046875' disk./.usagePercent‚77.959381557889 disk.sda3.out‚ 229.1796875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.0546875& disk.nvme0n1.out‚181691.5869140625 disk.sda1.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 711.1953125 network.sent‚ 1.1467556e+07 disk.sda.in‚0›N2å‚(:… ¸ÿçÔÏžŠy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.526' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3052 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yqZ(œæ‚(:‘ ÇÿçÔñÉÁw gpu.0.gpu‚36.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.773' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.75460000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ Â|€ç‚(:õ ÇÿçÔûú®x# disk.nvme0c0n1.in‚ 19554.0078125 disk.dm-1.out‚ 229.56640625 disk.sda.in‚0' proc.memory.availableMB‚ 711.1953125) proc.memory.percent‚19.92175579071045 proc.cpu.threads‚17 network.sent‚ 1.1473885e+07 network.recv‚ 2.435669e+06) disk./.usagePercent‚77.96095651623965 disk.drbd1260.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚182013.0517578125 disk.dm-0.in‚ 19554.0078125# disk.dm-0.out‚182013.0517578125 disk.drbd1260.out‚ 1.0546875! disk.nvme0n1.in‚ 19554.0078125 disk.dm-1.in‚0 disk.sda.out‚ 229.56640625 disk.sda3.out‚ 229.56640625# proc.memory.rssMB‚ 3263.98046875 cpu‚25.16132340409743% disk./.usageGB‚1161.6745910644531& disk.nvme0n1.out‚182013.0517578125 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚95.65920829772949 disk.sda1.in‚0\/E´2è- _wandbruntime‚7604  _runtime‚7604°”Åé¿ % lr-AdamW/pg1‚0.000537760830551036 % lr-AdamW/pg2‚0.000537760830551036 trainer/global_step‚4199 % _timestamp‚1.7884773981554697e+09 _runtime‚7604.911359687 _step‚341/†GÀÊêÄ _runtime‚7604.911359687 _step‚341 % lr-AdamW/pg1‚0.000537760830551036 % lr-AdamW/pg2‚0.000537760830551036 trainer/global_step‚4199 % _timestamp‚1.7884773981554697e+09Õ(XÜþ2ë- _wandbruntime‚7604  _runtime‚7604wÓX& ì†P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105108335 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56115331 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚603060159221248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10230745022464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56115326 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38286050 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94402099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617139989067264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚617139863828992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _step‚342 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660544789023196 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚650335582826736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884773981570323e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚603060108018176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚676483771091456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10227134361600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666256663121408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38286768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚650335608254704 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105134675 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚666256663121408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666256636729856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88732335 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94402099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617139989067264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚617139863828992 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _step‚342 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚660544789023196 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚650335582826736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884773981570323e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚603060108018176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚676483771091456 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10227134361600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666256663121408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38286768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚650335608254704 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105134675 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚666256663121408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666256636729856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88732335 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56115331 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚603060159221248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10230745022464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56115326 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Ö”}©’î‚(:‡ ÖÿçÔô¯Ÿz gpu.0.gpu‚35.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚187.522' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.5044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëJsï‚(:è ÖÿçÔÃÀØ{ disk.sda.out‚229.9375 disk.sda3.in‚0# proc.memory.rssMB‚ 3263.98046875% disk./.usageGB‚1161.6755599975586 disk.sda1.out‚0( disk.nvme0c0n1.out‚182479.1962890625 disk.dm-1.out‚229.9375 disk.sda3.out‚229.9375 disk.drbd1260.in‚0& disk.nvme0n1.out‚182479.1962890625$ memory_percent‚95.66097259521484) disk./.usagePercent‚77.96102154215335# disk.nvme0c0n1.in‚ 19554.0078125 disk.sda.in‚0! disk.nvme0n1.in‚ 19554.0078125) proc.memory.percent‚19.92175579071045 network.recv‚ 2.438816e+06 disk.sda2.in‚0 disk.dm-0.in‚ 19554.0078125 disk.drbd1260.out‚ 1.05859375% proc.memory.availableMB‚ 710.90625 cpu‚25.16144101532437 proc.cpu.threads‚17 network.sent‚ 1.1478834e+07 disk.sda2.out‚0 disk.sda1.in‚0# disk.dm-0.out‚182479.1962890625 disk.dm-1.in‚0Ÿ‰Ý©2ð- _wandbruntime‚7606  _runtime‚7606œ¾˜¾'ñN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617282475833344 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10215221162638 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚676639959132160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193912033 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105132902 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617282603155456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚603199398407168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 trainer/global_step‚4199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94423039 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666410414288896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193885661 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105159268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14083077426176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666410440680448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105132902 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56128131 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚666410440680448 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚676643596197888 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660697524567618 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816j}‡Ä‹ò•N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193885661 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚650485937495840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56128136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88752765 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 _step‚343 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193912033 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚676643596197888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10229544843264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚676643596197888 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660697524567618 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617282475833344 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10215221162638 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚676639959132160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193912033 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105132902 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617282603155456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60319jŽ G9398407168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 trainer/global_step‚4199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94423039 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666410414288896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193885661 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105159268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14083077426176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666410440680448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105132902 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56128131 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚666410440680448 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684293187540480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚610024627467776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668181579835504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚657852044883536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95482339 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚673945514681856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673945514681856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106366377 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚713 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610024678670848 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚657852070311504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚673945541073408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38726758 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684293187540480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10347672858624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95481621 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚673945541073408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38726758 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684293187540480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10347672858624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95481621 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684293187540480 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚610024627467776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Ú“Ú2ˆ- _wandbruntime‚7705  _runtime‚7705*F¯'‰N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89773959 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚624413201458688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668334315379926 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106390970 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668337949470786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674099318632448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884774987668977e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196164935 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56768386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10331941255254 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89773965 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95503848 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95503137 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚684449375581184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56768381 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _runtime‚ 7705.52868116 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674099292240896 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10350083340288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14249359720960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89773965 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38735462 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚674099292240896 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196164935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚658002399552640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106390970 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚674099318632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38734756 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106364604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚624413328780800 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196138563 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10353694014464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚658002374124672 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10335549918146 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚684453012646912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106364604 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14249283601920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249¾+Ú'Š”N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106364604 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14249283601920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89773959 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚624413201458688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚668334315379926 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106390970 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668337949470786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674099318632448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884774987668977e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196164935 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56768386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10331941255254 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89773965 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95503848 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95503137 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚684449375581184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56768381 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _runtime‚ 7705.52868116 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674099292240896 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10350083340288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14249359720960 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89773965 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38735462 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚674099292240896 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196164935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚658002399552640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106390970 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚674099318632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38734756 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106364604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚624413328780800 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196138563 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10353694014464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚658002374124672 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10335549918146 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚684453012646912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532Ûç­æY2‹- _wandbruntime‚7705  _runtime‚7705Y,2\ŒÖ .loss/FRt1/assignment_loss‚1.43719396532559 1loss/FBt1/assignment_loss‚0.25803732766787735 2loss/SRqqt4/assignment_loss‚2.2234186058243117 1loss/FBt4/assignment_loss‚0.25869441023805057 /loss/FRt1/detection_loss‚1.6733958375751556 _step‚348 2loss/SRqqt3/assignment_loss‚1.9477764142211527 /loss/FRt3/detection_loss‚3.2622949935141063 /loss/FBt3/detection_loss‚2.3682533994317057 _runtime‚7705.533532279 1loss/SRqqt2/assignment_loss‚3.155553380648295 1loss/SRqqt3/detection_loss‚2.1150524593889712 /loss/FBt2/detection_loss‚0.8451769466337282 %loss/total_loss‚48.52915083871754 % _timestamp‚1.7884774987767816e+09 /loss/FRt3/assignment_loss‚4.348796526590983 0loss/FRt4/assignment_loss‚3.5320195303513455 0loss/FBt3/assignment_loss‚0.3563126833803835 1loss/SRqqt1/detection_loss‚0.9844269915794333 trainer/global_step‚4249 0loss/FRt2/assignment_loss‚3.0576984193589953 1loss/FBt2/assignment_loss‚0.23132301717851078 1loss/SRqqt2/detection_loss‚3.4397512127955756 /loss/FBt1/detection_loss‚1.9685515348727887 /loss/FBt4/detection_loss‚2.1361047625541687 2loss/SRqqt1/assignment_loss‚1.2793906830095996 epoch‚3 /loss/FRt2/detection_loss‚2.1321043911907407 .loss/FRt4/detection_loss‚3.297000974416733 0loss/SRqqt4/detection_loss‚2.220822370969332é¨0íaÛ 1loss/SRqqt2/assignment_loss‚3.155553380648295 1loss/SRqqt3/detection_loss‚2.1150524593889712 /loss/FBt2/detection_loss‚0.8451769466337282 %loss/total_loss‚48.52915083871754 % _timestamp‚1.7884774987767816e+09 /loss/FRt3/assignment_loss‚4.348796526590983 0loss/FRt4/assignment_loss‚3.5320195303513455 0loss/FBt3/assignment_loss‚0.3563126833803835 1loss/SRqqt1/detection_loss‚0.9844269915794333 trainer/global_step‚4249 0loss/FRt2/assignment_loss‚3.0576984193589953 1loss/FBt2/assignment_loss‚0.23132301717851078 1loss/SRqqt2/detection_loss‚3.4397512127955756 /loss/FBt1/detection_loss‚1.9685515348727887 /loss/FBt4/detection_loss‚2.1361047625541687 2loss/SRqqt1/assignment_loss‚1.2793906830095996 epoch‚3 /loss/FRt2/detection_loss‚2.1321043911907407 .loss/FRt4/detection_loss‚3.297000974416733 0loss/SRqqt4/detection_loss‚2.220822370969332 .loss/FRt1/assignment_loss‚1.43719396532559 1loss/FBt1/assignment_loss‚0.25803732766787735 2loss/SRqqt4/assignment_loss‚2.2234186058243117 1loss/FBt4/assignment_loss‚0.25869441023805057 /loss/FRt1/detection_loss‚1.6733958375751556 _step‚348 2loss/SRqqt3/assignment_loss‚1.9477764142211527 /loss/FRt3/detection_loss‚3.2622949935141063 /loss/FBt3/detection_loss‚2.3682533994317057 _runtime‚7705.533532279ÜÐza’Ž‚(:‡ ¿€èÔã½ x gpu.0.gpu‚46.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚197.984' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.5968 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eâšâf‚(:Û ¿€èÔ΀z% disk./.usageGB‚1162.1785583496094 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.66528797149658( proc.memory.availableMB‚ 710.19921875# proc.memory.rssMB‚ 3263.98046875$ disk.dm-0.out‚184750.54443359375 disk.dm-1.out‚233.25 disk.sda2.in‚0 cpu‚25.162803683507338 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 19830.375) disk.nvme0c0n1.out‚184750.54443359375 disk.dm-0.in‚ 19830.375 disk.drbd1260.in‚0' disk.nvme0n1.out‚184750.54443359375 disk.sda2.out‚0) proc.memory.percent‚19.92175579071045 network.sent‚ 1.1701885e+07( disk./.usagePercent‚77.9947781836032 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚233.25 disk.sda3.out‚233.25 disk.drbd1260.out‚ 1.0703125 disk.nvme0n1.in‚ 19830.375 network.recv‚ 2.471853e+06%ÓH¥‚(:… ΀èÔ€ÐÎw gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]/š]x‘‚(:í ΀èÔÒ›Ð~ disk.nvme0n1.in‚ 19830.375) disk.nvme0c0n1.out‚185097.53955078125 disk.dm-0.in‚ 19830.375$ disk.dm-0.out‚185097.53955078125 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.07421875 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 19830.375 disk.dm-1.out‚ 233.84765625 disk.sda.in‚0( proc.memory.availableMB‚ 709.88671875# proc.memory.rssMB‚ 3263.98046875 proc.cpu.threads‚17% disk./.usageGB‚1162.1785583496094' disk.nvme0n1.out‚185097.53955078125 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 233.84765625# memory_percent‚95.6671953201294) proc.memory.percent‚19.92175579071045 cpu‚25.162908596307965 network.sent‚ 1.1811137e+07( disk./.usagePercent‚77.9947781836032 disk.sda1.out‚0 disk.sda.out‚ 233.84765625 disk.sda3.in‚0 disk.drbd1260.in‚0 network.recv‚ 2.476805e+06Šý¥‘’‚(:† Ý€èÔ¯¾úy gpu.0.gpu‚28.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚121.444' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2888 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯v“‚(:ë Ý€èÔø¹ÿz' disk.nvme0n1.out‚185322.93408203125 disk.nvme0c0n1.in‚ 19830.375 disk.dm-0.in‚ 19830.375 disk.sda3.out‚ 234.3203125 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 19830.375 disk.sda1.out‚0) disk.nvme0c0n1.out‚185322.93408203125 disk.dm-1.out‚ 234.3203125' proc.memory.availableMB‚ 710.0078125) proc.memory.percent‚19.92175579071045 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.1817255e+07 disk.sda.out‚ 234.3203125$ disk.dm-0.out‚185322.93408203125 disk.dm-1.in‚0$ memory_percent‚95.66645622253418 cpu‚25.163067284301317 disk.drbd1260.out‚ 1.07421875 disk.sda1.in‚0# proc.memory.rssMB‚ 3263.98046875 proc.cpu.threads‚17 network.recv‚ 2.481437e+06) disk./.usagePercent‚78.02841168542909% disk./.usageGB‚1162.6797218322754 disk.sda.in‚0 disk.sda3.in‚0yoä’”‚(:‡ ì€èԚȵx gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.451' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.2902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)¹>Ö •‚(:è ì€èÔü•èz disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 234.8203125 disk.sda3.in‚0' disk.nvme0n1.out‚185717.95849609375' proc.memory.availableMB‚ 710.7890625 cpu‚25.163202682047327 disk.dm-1.out‚ 234.8203125 network.sent‚ 1.1820219e+07 network.recv‚ 2.484645e+06 disk.nvme0c0n1.in‚ 19830.375) disk.nvme0c0n1.out‚185717.95849609375$ disk.dm-0.out‚185717.95849609375 disk.sda3.out‚ 234.8203125 disk.drbd1260.out‚1.078125$ disk./.usageGB‚1162.6543nbmµg69354248 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 19830.375 disk.nvme0n1.in‚ 19830.375$ memory_percent‚95.66168785095215) proc.memory.percent‚19.92175579071045) disk./.usagePercent‚78.02671025935652 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3263.98046875 proc.cpu.threads‚17aéi@’–‚(:‡ û€èÔ•Üw gpu.0.gpu‚91.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚283.251' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.6502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Dæ}w—‚(:ì û€èÔúäÿy) disk.nvme0c0n1.out‚186234.22021484375 disk.sda3.in‚0 disk.sda3.out‚ 235.453125$ memory_percent‚95.66335678100586$ disk./.usageGB‚1161.833423614502$ disk.dm-0.out‚186234.22021484375 disk.sda.out‚ 235.453125 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 19968.765625 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 2.489525e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 235.453125& proc.memory.availableMB‚ 710.515625) proc.memory.percent‚19.92175579071045 cpu‚25.163055769397353" disk.nvme0c0n1.in‚ 19968.765625 disk.dm-0.in‚ 19968.765625 disk.dm-1.in‚0 disk.sda.in‚0 disk.drbd1260.out‚1.078125' disk.nvme0n1.out‚186234.22021484375# proc.memory.rssMB‚ 3263.98046875 proc.cpu.threads‚17 network.sent‚ 1.1826609e+07( disk./.usagePercent‚77.9716159019428qf­2˜- _wandbruntime‚7775  _runtime‚7775\NpÆ™À & lr-AdamW/pg1‚0.0005305219590973922 & lr-AdamW/pg2‚0.0005305219590973922 trainer/global_step‚4299 $ _timestamp‚1.788477569171026e+09 _runtime‚7775.926501494 _step‚349«­r˚Štrainer/global_step‚4299 $ _timestamp‚1.788477569171026e+09 _runtime‚7775.926501494 _step‚349 & lr-AdamW/pg1‚0.0005305219590973922 & lr-AdamW/pg2‚0.0005305219590973922Ýaÿç2›- _wandbruntime‚7775  _runtime‚7775ô (œ†P \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681634392633856 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681634392633856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚616989146917376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107598079 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _step‚350 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107598079 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681634419025408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107571739 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚675818370647812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692106241041920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96563051 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10453472370160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90774741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39167220 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90774741 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681634419025408 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107571739 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14411969489408 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631401040274432 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14411893357056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚7775.927691357 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692102603989504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665368506940336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10471822016512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681634419025408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107571739 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚675818370647812 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692106241041920 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96563051 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10453472370160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90774741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39167220 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90774741 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681634419025408 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107571739 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14411969489408 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚4299 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631401040274432 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14411893357056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚7775.927691357 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692102603989504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚665368506940336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10471822016512 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198417837 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚681788170192896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617128437306368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692262429095936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10474232511488 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _step‚351 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57408636 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚681788170192896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10455878673654 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10474232511488 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10452270010762 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _runtime‚7776.943706156 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681788196584448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57408631 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107622672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10470621837312 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107596306 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚692258792030208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198417837 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚631543764070400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10470621837312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665518836181472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96583868 % _timestamp‚1.7884775701875162e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198391465 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631543636748288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198391465 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚675971106192234 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90795165 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚692262429095936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198391465 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚675971106192234 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90795165 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚692262429095936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198417837 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚681788170192896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617128437306368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692262429095936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10474232511488 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _step‚351 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57408636 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚681788170192896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10455878673654 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10474232511488 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10452270010762 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 _runtime‚7776.943706156 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681788196584448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57408631 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107622672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚697 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10470621837312 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107596306 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚692258792030208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198417837 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚631543764070400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10470621837312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665518836181472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96583868 % _timestamp‚1.7884775701875162e+09ßÊ*ÈÇ2¡- _wandbruntime‚7776  _runtime‚7776Cý ŽU¢Ï .loss/FBt1/detection_loss‚3.047235051790873 _runtime‚7776.946337004 trainer/global_step‚4299 % _timestamp‚1.7884775701906607e+09 /loss/FRt3/assignment_loss‚5.426452040672302 2loss/SRqqt1/assignment_loss‚1.1582687269726937 0loss/SRqqt4/detection_loss‚2.272304981946945 /loss/FBt2/detection_loss‚0.8724213818460703 .loss/FBt3/detection_loss‚2.822402496086924 /loss/FRt2/assignment_loss‚2.771276548260548 1loss/SRqqt2/assignment_loss‚3.511589845021566 1loss/FBt2/assignment_loss‚0.21883846689015182 /loss/FBt4/assignment_loss‚0.535090479219798 -loss/FRt4/detection_loss‚2.76012966803142 /loss/FRt1/detection_loss‚1.5819049559489728 /loss/FRt2/detection_loss‚1.9238619187571964 1loss/SRqqt3/detection_loss‚3.4149125926196575 0loss/FBt3/assignment_loss‚0.3655972751837812 .loss/FRt3/detection_loss‚3.683722034096718 /loss/FBt4/detection_loss‚2.6244666546583177 /loss/FRt4/assignment_loss‚3.771495543207441 0loss/FBt1/assignment_loss‚1.3003409142709441 1loss/SRqqt1/detection_loss‚0.9749910814179615 &loss/total_loss‚54.822850684207445 epoch‚3 2loss/SRqqt4/assignment_loss‚1.6522205877853067 0loss/SRqqt2/detection_loss‚4.226993987957637 0loss/FRt1/assignment_loss‚2.0967842288639233 2loss/SRqqt3/assignment_loss‚1.8095492227002978 _step‚352Ë>1Z£Ô epoch‚3 2loss/SRqqt4/assignment_loss‚1.6522205877853067 0loss/SRqqt2/detection_loss‚4.226993987957637 0loss/FRt1/assignment_loss‚2.0967842288639233 2loss/SRqqt3/assignment_loss‚1.8095492227002978 _step‚352 .loss/FBt1/detection_loss‚3.047235051790873 _runtime‚7776.946337004 trainer/global_step‚4299 % _timestamp‚1.7884775701906607e+09 /loss/FRt3/assignment_loss‚5.426452040672302 2loss/SRqqt1/assignment_loss‚1.1582687269726937 0loss/SRqqt4/detection_loss‚2.272304981946945 /loss/FBt2/detection_loss‚0.8724213818460703 .loss/FBt3/detection_loss‚2.822402496086924 /loss/FRt2/assignment_loss‚2.771276548260548 1loss/SRqqt2/assignment_loss‚3.511589845021566 1loss/FBt2/assignment_loss‚0.21883846689015182 /loss/FBt4/assignment_loss‚0.535090479219798 -loss/FRt4/detection_loss‚2.76012966803142 /loss/FRt1/detection_loss‚1.5819049559489728 /loss/FRt2/detection_loss‚1.9238619187571964 1loss/SRqqt3/detection_loss‚3.4149125926196575 0loss/FBt3/assignment_loss‚0.3655972751837812 .loss/FRt3/detection_loss‚3.683722034096718 /loss/FBt4/detection_loss‚2.6244666546583177 /loss/FRt4/assignment_loss‚3.771495543207441 0loss/FBt1/assignment_loss‚1.3003409142709441 1loss/SRqqt1/detection_loss‚0.9749910814179615 &loss/total_loss‚54.822850684207445à7k7¤’¤‚(:‡ ŠèÔˆÏÕx gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚268.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ð¾\{¥‚(:ð ŠèÔñÌ‹y disk.sda.out‚ 235.796875 disk.drbd1260.out‚ 1.08203125' disk.nvme0n1.out‚186497.97705078125 disk.sda1.in‚0) disk.nvme0c0n1.out‚186497.97705078125) proc.memory.percent‚19.92175579071045 proc.cpu.threads‚17 network.recv‚ 2.493116e+06% disk./.usageGB‚1161.8100509643555$ disk.dm-0.out‚186497.97705078125 disk.dm-1.out‚ 235.796875 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3263.98046875 network.sent‚ 1.1831538e+07 disk.dm-0.in‚ 19968.765625 disk.sda3.in‚0 disk.sda3.out‚ 235.796875 disk.nvme0n1.in‚ 19968.765625" disk.nvme0c0n1.in‚ 19968.765625$ memory_percent‚95.66483497619629' proc.memory.availableMB‚ 710.2734375) disk./.usagePercent‚77.97004734378051 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 cpu‚25.16287834640643 ‘¦‚(:† ™èÔǤ¾w gpu.0.gpu‚54.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚117.293' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4586 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U€á(:ö ™èÔ¨”†x disk.drbd1260.out‚ 1.08203125' disk.nvme0n1.out‚186858.69580078125 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 19968.765625) disk.nvme0c0n1.out‚186858.69580078125 disk.dm-0.in‚ 19968.765625 disk.dm-1.in‚0 network.sent‚ 1.1940295e+07) disk./.usagePercent‚77.97162511821404% disk./.usageGB‚1161.8335609436035 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 236.23046875$ memory_percent‚95.66507339477539) proc.memory.percent‚19.92175579071045 network.recv‚ 2.499986e+06 disk.sda.in‚0 disk.drbd1260.in‚0 cpu‚25.162765692349154 proc.cpu.threads‚17 disk.sda.out‚ 236.23046875 disk.sda3.out‚ 236.23046875 disk.nvme0n1.in‚ 19968.765625 disk.sda1.out‚0$ disk.dm-0.out‚186858.69580078125& proc.memory.availableMB‚ 710.234375# proc.memory.rssMB‚ 3263.98046875Àüi¨‚(:’ ¨èÔäê”w gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚251.099' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.219800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dPŽ°©‚(:ô ¨èÔøþ¦x disk.drbd1260.in‚0 cpu‚25.162613472503715 proc.cpu.threads‚17' disk.nvme0n1.out‚187050.54833984375" disk.nvme0c0n1.in‚ 19968.765625$ disk.dm-0.out‚187050.54833984375 disk.sda.in‚0 disk.sda.out‚ 236.67578125 disk.sda3.out‚ 236.67578125! proc.memory.rssMB‚ 3263.984375 network.sent‚ 1.1945127e+07 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 236.67578125 disk.drbd1260.out‚ 1.0859375 network.recv‚ 2.503062e+06 disk.nvme0n1.in‚ 19968.765625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0$ memory_percent‚95.66206932067871' proc.memory.availableMB‚ 710.7265625) proc.memory.percent‚19.92177963256836) disk./.usagePercent‚77.98915549012868% disk./.usageGB‚1162.0947761535645 disk.sda2.in‚0) disk.nvme0c0n1.out‚187050.54833984375 disk.dm-0.in‚ 19968.765625Qª0¾†ª‚(:û ·èÔ¸ ¶z" disk.nvme0n1.in‚20082.48828125' disk.nvme0n1.out‚187267.11083984375 disk.sda2.out‚0 network.sent‚ 1.1951457e+07) disk./.usagePercent‚77.98901775807525% disk./.usageGB‚1162.0927238464355) disk.nvme0c0n1.out‚187267.11083984375 disk.drbd1260.in‚0) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17 disk.dm-0.in‚20082.48828125$ disk.dm-0.out‚187267.11083984375 disk.dm-1.in‚0 disk.sda.out‚ 237.10546875 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚95.66512107849121' proc.memory.availableMB‚ 710.2265625 network.recv‚ 2.507706e+06 disk.sda3.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.98828125 cpu‚25.16292259828615$ disk.nvme0c0n1.in‚20082.48828125 disk.dm-1.out‚ 237.10546875 disk.sda.in‚0 disk.sda3.out‚ 237.10546875 disk.drbd1260.out‚ 1.0859375ý¿ ב«‚(:† ·èÔÇœÛz gpu.0.gpu‚42.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.107' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6214 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%cQA2¬- _wandbruntime‚7841  _runtime‚7841 <±jÇ­Á % _timestamp‚1.7884776349792185e+09 _runtime‚7841.735106266 _step‚353 & lr-AdamW/pg1‚0.0005268381877458363 & lr-AdamW/pg2‚0.0005268381877458363 trainer/global_step‚4349Á¦›Ì®Æ trainer/global_step‚4349 % _timestamp‚1.7884776349792185e+09 _runtime‚7841.735106266 _step‚353 & lr-AdamW/pg1‚0.0005268381877458363 & lr-AdamW/pg2‚0.0005268381877458363áQÈ 2¯- _wandbruntime‚7841  _runtime‚7841ðR8 (°†P YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚699915657490944 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200625722 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108803441 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638531392718848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚623953717570048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14577726351872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91795935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39607486 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 _runtime‚7841.736622494 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10588749852672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚689323296977408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚689323296977408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39606768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638531520054272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚683458795550772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚683455161460120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58036076 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58036081 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚623953666366976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 trainer/global_step‚4349 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108829781 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689323270585856 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97643567 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91795941 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10588749852672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10570192462984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10592360513536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672884994425104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689323270585856 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚699912020438528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10573801125668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200599376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚699912020438528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0–S’ý±‹P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚699915657490944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚699912020438528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10573801125668 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200599376 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚699912020438528 60Dº…IþeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚699915657490944 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200625722 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108803441 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638531392718848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚623953717570048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14577726351872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91795935 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39607486 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 _runtime‚7841.736622494 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10588749852672 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚689323296977408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚689323296977408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39606768 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638531520054272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚683458795550772 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚683455161460120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58036076 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58036081 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚623953666366976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 trainer/global_step‚4349 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108829781 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689323270585856 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97643567 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91795941 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10588749852672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10570192462984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10592360513536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672884994425104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689323270585856 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200670739 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚700071845544960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10594771008512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14581132674048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97664398 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200644367 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91816359 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108854374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108828008 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200644367 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚689477074536448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚689477048144896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683611531095402 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91816365 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14581056555008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200670739 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 $ _timestamp‚1.788477636006274e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚689477048144896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673035298238272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚700071845544960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚624093007959040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673035323666240 trainer/global_step‚4349 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108854374 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719ë[S'´”N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14581056555008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200670739 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 $ _timestamp‚1.788477636006274e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚689477048144896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673035298238272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚700071845544960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚624093007959040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673035323666240 trainer/global_step‚4349 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108854374 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10591160334336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58048886 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10572598766270 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 _runtime‚7842.762485212 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10591160334336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10594771008512 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638674013310976 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200670739 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚700071845544960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10594771008512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14581132674048 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97664398 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200644367 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91816359 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108854374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108828008 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200644367 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚689477074536448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚689477048144896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683611531095402 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91816365 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ã%áñÁ2µ- _wandbruntime‚7842  _runtime‚7842W­$/W¶Ñ 1loss/FBt3/assignment_loss‚0.02642851001486027 1loss/SRqqt1/detection_loss‚0.9902366656532957 _runtime‚7842.764701229 1loss/SRqqt3/assignment_loss‚2.058213179630618 2loss/SRqqt4/assignment_loss‚1.7133075629289334 epoch‚3 trainer/global_step‚4349 1loss/SRqqt1/assignment_loss‚1.186767792275314 1loss/SRqqt2/assignment_loss‚2.116936961809794 0loss/FBt1/assignment_loss‚0.3121738234507835 0loss/SRqqt4/detection_loss‚2.311238034413411 /loss/FBt1/detection_loss‚3.0549187319619313 _step‚356 0loss/SRqqt3/detection_loss‚2.258823340938937 .loss/FBt3/detection_loss‚2.791840847333272 /loss/FBt4/detection_loss‚2.6149404599116397 % _timestamp‚1.7884776360093732e+09 0loss/FRt1/assignment_loss‚2.0797302969296774 1loss/FBt2/assignment_loss‚0.16679623262353194 .loss/FRt2/detection_loss‚2.308870256726825 .loss/FRt3/detection_loss‚3.477483768212168 /loss/FBt2/detection_loss‚0.7022143889445107 0loss/FRt4/assignment_loss‚3.6812620252370833 0loss/FBt4/assignment_loss‚0.2488770010198865 /loss/FRt4/detection_loss‚2.7461378902196882 %loss/total_loss‚49.57112311565288 /loss/FRt1/detection_loss‚1.6516912085826818 0loss/SRqqt2/detection_loss‚4.162003288666408 0loss/FRt2/assignment_loss‚2.5446847095045935 /loss/FRt3/assignment_loss‚4.365546138663041”…4Ÿ\·Ö 0loss/FRt1/assignment_loss‚2.0797302969296774 1loss/FBt2/assignment_loss‚0.16679623262353194 .loss/FRt2/detection_loss‚2.308870256726825 .loss/FRt3/detection_loss‚3.477483768212168 /loss/FBt2/detection_loss‚0.7022143889445107 0loss/FRt4/assignment_loss‚3.6812620252370833 0loss/FBt4/assignment_loss‚0.2488770010198865 /loss/FRt4/detection_loss‚2.7461378902196882 %loss/total_loss‚49.57112311565288 /loss/FRt1/detection_loss‚1.6516912085826818 0loss/SRqqt2/detection_loss‚4.162003288666408 0loss/FRt2/assignment_loss‚2.5446847095045935 /loss/FRt3/assignment_loss‚4.365546138663041 1loss/FBt3/assignment_loss‚0.02642851001486027 1loss/SRqqt1/detection_loss‚0.9902366656532957 _runtime‚7842.764701229 1loss/SRqqt3/assignment_loss‚2.058213179630618 2loss/SRqqt4/assignment_loss‚1.7133075629289334 epoch‚3 trainer/global_step‚4349 1loss/SRqqt1/assignment_loss‚1.186767792275314 1loss/SRqqt2/assignment_loss‚2.116936961809794 0loss/FBt1/assignment_loss‚0.3121738234507835 0loss/SRqqt4/detection_loss‚2.311238034413411 /loss/FBt1/detection_loss‚3.0549187319619313 _step‚356 0loss/SRqqt3/detection_loss‚2.258823340938937 .loss/FBt3/detection_loss‚2.791840847333272 /loss/FBt4/detection_loss‚2.6149404599116397 % _timestamp‚1.7884776360093732e+09äE5Ê}¸‚(:ò ÆèÔíÓìx disk.sda1.out‚0$ proc.memory.availableMB‚710.4375 cpu‚25.162845516292045$ disk.nvme0c0n1.in‚20107.98046875 disk.dm-0.in‚20107.98046875# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17 network.recv‚ 2.511032e+06) disk./.usagePercent‚77.98074589863641$ disk./.usageGB‚1161.969467163086( disk.nvme0c0n1.out‚187677.2666015625# disk.dm-0.out‚187677.2666015625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.08984375& disk.nvme0n1.out‚187677.2666015625 disk.dm-1.in‚0 disk.dm-1.out‚ 237.5703125 disk.sda.out‚ 237.5703125 disk.sda3.out‚ 237.5703125" disk.nvme0n1.in‚20107.98046875 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚95.66383361816406 network.sent‚ 1.196946e+07 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.out‚0¶éä𒹂(:‡ ÆèÔÛÊÂy gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚269.364' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.8728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@XÄ‚º‚(:÷ ÕèÔíä·x disk.sda3.in‚0 disk.sda3.out‚ 238.25390625( disk.nvme0c0n1.out‚188189.8369140625) proc.memory.percent‚19.92180347442627 cpu‚25.162729824908517 proc.cpu.threads‚17 disk.drbd1260.in‚0" disk.nvme0n1.in‚20107.98046875& disk.nvme0n1.out‚188189.8369140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0$ memory_percent‚95.66383361816406 disk.sda1.out‚0 disk.dm-0.in‚20107.98046875 disk.dm-1.out‚ 238.25390625 disk.sda.out‚ 238.25390625 network.sent‚ 1.2065085e+07 network.recv‚ 2.517724e+06 disk.drbd1260.out‚ 1.08984375 disk.sda1.in‚0$ disk.nvme0c0n1.in‚20107.98046875# disk.dm-0.out‚188189.8369140625$ proc.memory.availableMB‚710.4375# proc.memory.rssMB‚ 3263.98828125) disk./.usagePercent‚77.99291060864091% disk./.usageGB‚1162.1507301330566ÂÅ mœ»‚(:‘ ÕèÔþì±x gpu.0.gpu‚93.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚288.535' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.70700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àã ·;¼‚(:ù äèÔ—ºÓx disk.sda3.out‚ 238.78515625) proc.memory.percent‚19.92180347442627% disk./.usageGB‚1162.1504173278809" disk.nvme0n1.in‚20107.98046875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚20107.98046875 disk.sda.out‚ 238.78515625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.09765625 disk.sda1.out‚0 disk.dm-0.in‚20107.98046875$ memory_percent‚95.66230773925781$ proc.memory.availableMB‚710.6875# proc.memory.rssMB‚ 3263.98828125 cpu‚25.162798478297827 proc.cpu.threads‚17 network.sent‚ 1.2069111e+07 network.recv‚ 2.520162e+06 disk.sda2.in‚0) disk.nvme0c0n1.out‚188655.50732421875$ disk.dm-0.out‚188655.50732421875 disk.dm-1.in‚0 disk.sda.in‚0( disk./.usagePercent‚77.9928896160231' disk.nvme0n1.out‚¤°I188655.50732421875 disk.sda2.out‚0 disk.dm-1.out‚ 238.78515625/âí2½‚(:… äèÔ­òáx gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7546 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë`Š8¾‚(:ö óèÔ‘ù»x$ disk.nvme0c0n1.in‚20107.98046875) disk.nvme0c0n1.out‚188655.76123046875 disk.sda.out‚ 239.1953125 disk.sda2.out‚0# proc.memory.rssMB‚ 3263.98828125 cpu‚25.16286972451536 proc.cpu.threads‚17 network.recv‚ 2.524619e+06) disk./.usagePercent‚77.99288833598543% disk./.usageGB‚1162.1503982543945 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 239.1953125 disk.sda2.in‚0# memory_percent‚95.6624984741211 network.sent‚ 1.2075495e+07$ disk.dm-0.out‚188655.76123046875 disk.dm-1.out‚ 239.1953125 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚19.92180347442627 disk.dm-0.in‚20107.98046875 disk.drbd1260.out‚ 1.09765625" disk.nvme0n1.in‚20107.98046875' disk.nvme0n1.out‚188655.76123046875% proc.memory.availableMB‚ 710.65625syÑ¿‚(:… óèÔ¼ä›y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÿ¥û‡À‚(:ü ‚‚èÔù¢õw disk.sda2.in‚0 disk.dm-0.in‚20238.93359375 disk.dm-1.in‚0) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17) disk./.usagePercent‚77.99241702611491 disk.sda.out‚ 239.58203125 disk.sda3.in‚0' disk.nvme0n1.out‚188853.54345703125 disk.sda1.out‚0$ disk.nvme0c0n1.in‚20238.93359375$ disk.dm-0.out‚188853.54345703125' proc.memory.availableMB‚ 710.8984375 network.recv‚ 2.528206e+06 disk.sda.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚20238.93359375 disk.sda2.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚188853.54345703125 cpu‚25.162942795438216% disk./.usageGB‚1162.1433753967285 disk.sda3.out‚ 239.58203125 disk.drbd1260.out‚ 1.1015625 disk.dm-1.out‚ 239.58203125$ memory_percent‚95.66102027893066# proc.memory.rssMB‚ 3263.98828125 network.sent‚ 1.2080513e+07QL]–Á‚(:’ ‚‚èÔˆ«x gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚263.748' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.749599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HXå2Â- _wandbruntime‚7919  _runtime‚7919å©ÙœÇÃÁ & lr-AdamW/pg1‚0.0005231125466058334 & lr-AdamW/pg2‚0.0005231125466058334 trainer/global_step‚4399 % _timestamp‚1.7884777122883968e+09 _runtime‚7919.044684618 _step‚357ÃÖ8"ÌÄÆ & lr-AdamW/pg2‚0.0005231125466058334 trainer/global_step‚4399 % _timestamp‚1.7884777122883968e+09 _runtime‚7919.044684618 _step‚357 & lr-AdamW/pg1‚0.0005231125466058334å³Ç2ÿ2Å- _wandbruntime‚7919  _runtime‚7919%ã (ƆP VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110061483 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110035143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58676331 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40047454 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 _runtime‚7919.047030149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚691095586363080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚691091952272428 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚740 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98723780 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202852278 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92817135 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707725073939968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697012148537856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚735 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92817141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14743656367104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚680401456481904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10712899010560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 % _timestamp‚1.7884777122909586e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697012174929408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92817135 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚680401431053936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707725073939968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92817141 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58676326 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10694129881176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10690521218492 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707721436887552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202878624 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚740 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98723780 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202852278 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92817135 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707725073939968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697012148537856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚735 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92817141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14743656367104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚680401456481904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10712899010560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 % _timestamp‚1.7884777122909586e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697012174929408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92817135 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚680401431053936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚707725073939968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92817141 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58676326 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10694129881176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10690521218492 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707721436887552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202878624 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202923641 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40056183 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40056911 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92837559 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚733 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 % _timestamp‚1.7884777137429147e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10715309505536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚697165952488448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110086076 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚691244687816850 _step‚359 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707881261993984 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 _runtime‚7920.499543787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707881261993984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10711698831360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631057527408640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10692927521778 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98745314 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58689131 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14746950882304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92837565 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10711698831360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697165952488448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10715309505536 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202923641 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_Þ˜?Fû batch_end/inactive_split_bytes.all.allocated‚645804554409984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98746047 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10696536184670 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110086076 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110059710 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚645804427087872 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0I»ÙV'Ì•N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚645804554409984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98746047 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10696536184670 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110086076 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110059710 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚645804427087872 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92837559 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚680551760295072 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202897269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631057476205568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202923641 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40056183 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40056911 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92837559 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚733 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 % _timestamp‚1.7884777137429147e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10715309505536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚697165952488448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110086076 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚691244687816850 _step‚359 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707881261993984 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 _runtime‚7920.499543787 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707881261993984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10711698831360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631057527408640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10692927521778 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98745314 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58689131 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14746950882304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92837565 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10711698831360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697165952488448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10715309505536 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202923641 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0çÝ.†2Í- _wandbruntime‚7920  _runtime‚7920œ‰"ïZÎÔ /loss/FRt1/assignment_loss‚2.006651932702345 0loss/FRt3/assignment_loss‚4.5655942228105335 0loss/FBt3/assignment_loss‚0.8598783179289003 /loss/FRt1/detection_loss‚1.6640266233620296 -loss/FRt4/detection_loss‚2.70073369929665 %loss/total_loss‚47.92854963896273 2loss/SRqqt3/assignment_loss‚2.1615539257590837 .loss/FBt4/detection_loss‚2.337023515254259 % _timestamp‚1.7884777137449121e+09 /loss/FRt2/assignment_loss‚2.424748669220851 0loss/FRt4/assignment_loss‚3.4430180317477177 0loss/FBt2/assignment_loss‚0.1732487360763193 .loss/FRt3/detection_loss‚3.512586855226093 _runtime‚7920.500267341 2loss/SRqqt2/assignment_loss‚2.3464095312006332 2loss/SRqqt4/assignment_loss‚1.5740672912448646 1loss/SRqqt1/detection_loss‚1.0480498246375531 1loss/SRqqt3/detection_loss‚2.4862351135627643 epoch‚3 _step‚360 /loss/FRt2/detection_loss‚1.8143219161778688 1loss/SRqqt2/detection_loss‚3.2234181376064526 2loss/SRqqt1/assignment_loss‚1.2109857485555704 0loss/SRqqt4/detection_loss‚2.386376056075096 /loss/FBt3/detection_loss‚2.5168279707431793 2loss/FBt1/assignment_loss‚0.011171305276548082 0loss/FBt4/assignment_loss‚0.5085096049703566 /loss/FBt1/detection_loss‚2.0799924209713936 /loss/FBt2/detection_loss‚0.8731201885556671 trainer/global_step‚4399ŒÝn_ÏÙ /loss/FRt2/assignment_loss‚2.424748669220851 0loss/FRt4/assignment_loss‚3.4430180317477177 0loss/FBt2/assignment_loss‚0.1732487360763193 .loss/FRt3/detection_loss‚3.512586855226093 _runtime‚7920.500267341 2loss/SRqqt2/assignment_loss‚2.3464095312006332 2loss/SRqqt4/assignment_loss‚1.5740672912448646 1loss/SRqqt1/detection_loss‚1.0480498246375531 1loss/SRqqt3/detection_loss‚2.4862351135627643 epoch‚3 _step‚360 /loss/FRt2/detection_loss‚1.8143219161778688 1loss/SRqqt2/detection_loss‚3.2234181376064526 2loss/SRqqt1/assignment_loss‚1.2109857485555704 0loss/SRqqt4/detection_loss‚2.386376056075096 /loss/FBt3/detection_loss‚2.5168279707431793 2loss/FBt1/assignment_loss‚0.011171305276548082 0loss/FBt4/assignment_loss‚0.5085096049703566 /loss/FBt1/detection_loss‚2.0799924209713936 /loss/FBt2/detection_loss‚0.8731201885556671 trainer/global_step‚4399 /loss/FRt1/assignment_loss‚2.006651932702345 0loss/FRt3/assignment_loss‚4.5655942228105335 0loss/FBt3/assignment_loss‚0.8598783179289003 /loss/FRt1/detection_loss‚1.6640266233620296 -loss/FRt4/detection_loss‚2.70073369929665 %loss/total_loss‚47.92854963896273 2loss/SRqqt3/assignment_loss‚2.1615539257590837 .loss/FBt4/detection_loss‚2.337023515254259 % _timestamp‚1.7884777137449121e+09èYe4ےЂ(:‡  ‚èÔêÓÇy gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚242.607' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.5214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á•_}Ñ‚(:ò  ‚èÔèÈÍz network.recv‚ 2.537949e+06% disk./.usageGB‚1162.2167358398438" disk.nvme0n1.in‚20238.93359375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 240.28125 cpu‚25.16321940954429 disk.dm-0.in‚20238.93359375$ disk.dm-0.out‚189609.67919921875 disk.sda.out‚ 240.28125 disk.sda3.in‚0 disk.drbd1260.out‚ 1.10546875# proc.memory.rssMB‚ 3263.98828125) disk./.usagePercent‚77.99734030700587 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17' disk.nvme0n1.out‚189609.67919921875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚20238.93359375) disk.nvme0c0n1.out‚189609.67919921875 disk.sda.in‚0 disk.sda3.out‚ 240.28125# memory_percent‚95.6627368927002' proc.memory.availableMB‚ 710.6171875 network.sent‚ 1.2182801e+07f Õº‰Ò‚(:þ ¯‚èÔ™à€y network.recv‚ 2.544271e+06$ disk.dm-0.out‚190028.37451171875 disk.sda.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚20238.93359375' disk.nvme0n1.out‚190028.37451171875( proc.memory.availableMB‚ 710.62109375 cpu‚25.163391590672056 network.sent‚ 1.2280957e+07% disk./.usageGB‚1162.3812942504883$ disk.nvme0c0n1.in‚20238.93359375 disk.sda3.in‚0 disk.drbd1260.out‚ 1.10546875$ memory_percent‚95.66271305084229 proc.cpu.threads‚17 disk.sda3.out‚ 240.77734375) disk./.usagePercent‚78.00838396001794 disk.dm-0.in‚20238.93359375 disk.dm-1.in‚0 disk.dm-1.out‚ 240.77734375 disk.sda.out‚ 240.77734375 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.98828125) disk.nvme0c0n1.out‚190028.37451171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.percent‚19.92180347442627}4ú’Ó‚(:‡ ¯‚èÔ–®ªy gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚214.224' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.8448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0usBÔ‚(:… ¾‚èÔËûx gpu.0.gpu‚35.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.627 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%,2óxÕ‚(:í ¾‚èÔ²±çz% disk./.usageGB‚1162.4042472839355 disk.sda2.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 20360.1875) disk.nvme0c0n1.out‚190374.06396484375 disk.sda.in‚0 disk.sda.out‚ 241.1953125 disk.sda3.out‚ 241.1953125 disk.sda2.out‚0$ disk.dm-0.out‚190374.06396484375 disk.drbd1260.out‚1.109375$ memory_percent‚95.66271305084229# proc.memory.rssMB‚ 3263.98828125 proc.cpu.threads‚17) disk./.usagePercent‚78.00992435735145 disk.sda1.out‚0 disk.dm-1.out‚ 241.1953125 disk.sda3.in‚0 disk.nvme0n1.in‚ 20360.1875( proc.memory.availableMB‚ 710.62109375 network.sent‚ 1.2286004e+07 network.recv‚ 2.547847e+06 disk.dm-0.in‚ 20360.1875 disk.dm-1.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚190374.06396484375) proc.memory.percent‚19.92180347442627 cpu‚25.1635410615476654UvÖ‚(:… Í‚èÔ£õôx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c\ú¢tׂ(:é Í‚èÔ ë¾y disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.02677042112707 disk.dm-1.in‚0 disk.dm-1.out‚ 241.515625 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚19.92180347442627 cpu‚25.163772037296702% disk./.usageGB‚1162.6552658081055) disk.nvme0c0n1.out‚190624.51708984375$ disk.dm-0.out‚190624.51708984375 disk.sda.in‚0 disk.sda.out‚ 241.515625 disk.nvme0n1.in‚ 20360.1875( proc.memory.availableMB‚ 710.11328125# proc.memory.rssMB‚ 3263.98828125 network.sent‚ 1.2292239e+07 network.recv‚ 2.552606e+06 disk.nvme0c0n1.in‚ 20360.1875 disk.dm-0.in‚ 20360.1875 disk.sda3.out‚ 241.515625 disk.drbd1260.out‚1.109375' disk.nvme0n1.out‚190624.51708984375# memory_percent‚95.6658124923706 disk.sda3.in‚03ÕHÈ’Ø‚(:‡ Ü‚èÔ¦Úéx gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fæ[ñyÙ‚(:î Ü‚èÔ·ý€z disk.sda3.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚191005.35693359375$ memory_percent‚95.66426277160645 network.sent‚ 1.2297111e+07 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 cpu‚25.163928628252965 disk.sda3.out‚ 241.8515625) disk./.usagePercent‚78.02512710876478% disk./.usageGB‚1162.6307792663574 disk.dm-0.in‚ 20360.1875 disk.sda.in‚0' disk.nvme0n1.out‚191005.35693359375 disk.nvme0c0n1.in‚ 20360.1875# proc.memory.rssMB‚ 3263.98828125 network.recv‚ 2.556272e+06 disk.sda.out‚ 241.8515625 disk.nvme0n1.in‚ 20360.1875 disk.sda1.out‚0' proc.memory.availableMB‚ 710.3671875) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.11328125$ disk.dm-0.out‚191005.35693359375 disk.dm-1.in‚0 disk.dm-1.out‚ 241.8515625K“8nÚ‚(:ã ë‚èÔ´·x network.sent‚ 1.2303493e+07 network.recv‚ 2.561055e+06% disk./.usageGB‚1162.6307792663574 disk.drbd1260.out‚ 1.11328125) disk.nvme0c0n1.out‚191519.15380859375$ disk.dm-0.out‚191519.15380859375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 20360.1875' disk.nvme0n1.out‚191519.15380859375 disk.sda1.out‚0 disk.dm-1.out‚242.5625 disk.sda3.out‚242.5625# memory_percent‚95.6627368927002# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 20360.1875 disk.sda.in‚0 disk.sda.out‚242.5625 proc.cpu.threads‚17) disk./.usagePercent‚78.02512710876478 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 20360.1875' proc.memory.availableMB‚ 710.6171875 cpu‚25.16413018057235YúPœÛ‚(:‘ ë‚èÔç® x gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚276.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.372600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¢+q2Ü- _wandbruntime‚8013  _runtime‚8013©ýßÇÝÁ trainer/global_step‚4449 % _timestamp‚1.7884778068257236e+09 _runtime‚8013.581412697 _step‚361 & lr-AdamW/pg1‚0.0005193458120954127 & lr-AdamW/pg2‚0.0005193458120954127—TÌÞÆ & lr-AdamW/pg1‚0.0005193458120954127 & lr-AdamW/pg2‚0.0005193458120954127 trainer/global_step‚4449 % _timestamp‚1.7884778068257236e+09 _runtime‚8013.581412697 _step‚361éÕ²;2ß- _wandbruntime‚8013  _runtime‚8013Éë¤þ#"à…P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10829826846720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚715534490388992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10833437507584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652792246738944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 _runtime‚8013.582813527 _step‚362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704701052881408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93838341 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14909490269696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚698732377175388 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10810849974000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10829826846720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93838335 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 trainer/global_step‚4449 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111266845 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚715530853336576 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚637882756469248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205105180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99804635 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59316576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111293185 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚698728743084736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚687917893110736 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99803926 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59316581 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40488054 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14909416234496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788477806827142e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205131526 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704701052881408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40487350 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚704701026489856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10814458636684 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonÔDèitor.on_train_batch_start/inactive_split_bytes.all.freed‚652792121500672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚637882705266176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚715530853336576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111293185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10833437507584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187üþe(áŠP UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111293185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10833437507584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10829826846720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚715534490388992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10833437507584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652792246738944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 _runtime‚8013.582813527 _step‚362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704701052881408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93838341 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14909490269696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚698732377175388 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10810849974000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10829826846720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93838335 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 trainer/global_step‚4449 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111266845 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚715530853336576 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚637882756469248 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205105180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99804635 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59316576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111293185 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚698728743084736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚687917893110736 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99803926 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59316581 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40488054 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14909416234496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788477806827142e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205131526 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704701052881408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40487350 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚704701026489856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10814458636684 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652792121500672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚637882705266176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚715530853336576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912êrpoH2â- _wandbruntime‚8014  _runtime‚8014m·'ãN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111317778 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111291412 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10835848002560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205176543 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚715690678443008 % _timestamp‚1.7884778080151267e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205176543 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111291412 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93858759 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10813256277286 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚704854830440448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99825586 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10816864940178 _runtime‚8014.772319195 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10832237328384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14912752990208 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59329381 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10835848002560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205150171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚715687041377280 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚704854830440448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚688068247779840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14912824914944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205150171 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _step‚363 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93858759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40496887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652934871773184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638022046858240 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93858759 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10813256277286 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚704854830440448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99825586 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10816864940178 _runtime‚8014.772319195 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10832237328384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14912752990208 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59329381 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10835848002560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205150171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚715687041377280 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚704854830440448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚688068247779840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14912824914944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205150171 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _step‚363 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93858759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40496887 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚652934871773184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638022046858240 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10835848002560 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205176543 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚687 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚715690678443008 % _timestamp‚1.7884778080151267e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205176543 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111291412 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8108.511419783 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10950365343744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94859541 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚712389904441856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59956831 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94859535 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚723343906838016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚723340269785600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94859535 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644847275918848 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚695434380595504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644847224715776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112498547 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10934787392192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10953976004608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚723340269785600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10953976004608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94859541 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100885915 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15075591045120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112524887 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40928382 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 _step‚366 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207358082 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207384428 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187ÀFH(ùP LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706365533897044 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10931178729508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚712389904441856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚723340269785600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10953976004608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94859541 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100885915 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15075591045120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112524887 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40928382 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 _step‚366 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207358082 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207384428 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚707 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚712389930833408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884779017539926e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚8108.511419783 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10950365343744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94859541 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚712389904441856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59956831 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94859535 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚723343906838016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚723340269785600 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94859535 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644847275918848 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚695434380595504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644847224715776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112498547 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10934787392192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10953976004608 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94879965 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94879959 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10952775825408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚706518269441466 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚8111.346746971 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bTܽ0Åytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100907370 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚695584684408672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚723500094892032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660065553787904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 trainer/global_step‚4499 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚712543708392448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207429445 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59969631 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94879965 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚712543682000896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100906695 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59969636 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112523114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10952775825408 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112549480 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94879959 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10956386499584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10933585032794 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207403073 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15078913458176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚712543708392448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40937064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660065428562944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚695584709836640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112549480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112523114 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424¶)2M'ü—N ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚695584684408672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚723500094892032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660065553787904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 trainer/global_step‚4499 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚712543708392448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207429445 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59969631 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94879965 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚712543682000896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100906695 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59969636 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112523114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10952775825408 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112549480 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94879959 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10956386499584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10933585032794 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207403073 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15078913458176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚712543708392448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40937064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660065428562944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚695584709836640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112549480 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112523114 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94879965 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94879959 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10952775825408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚706518269441466 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚8111.346746971 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100907370 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136ï*W&2ý- _wandbruntime‚8111  _runtime‚8111•°KYþÓ 1loss/SRqqt4/detection_loss‚2.1880826821198336 .loss/FBt2/detection_loss‚0.867363193367101 %loss/total_loss‚49.91179714350245 /loss/FRt2/assignment_loss‚3.070923077125175 1loss/FBt1/assignment_loss‚0.46301404712524646 /loss/FRt1/detection_loss‚1.7182779138278215 /loss/FBt3/detection_loss‚2.5803766780429416 2loss/SRqqt2/assignment_loss‚2.2128779341777167 .loss/FBt4/detection_loss‚2.825647527972857 0loss/FBt3/assignment_loss‚0.3812128222327576 .loss/FRt2/detection_loss‚1.822872066176405 .loss/FRt4/detection_loss‚2.714433477537052 1loss/SRqqt1/detection_loss‚1.1064649063252634 2loss/SRqqt1/assignment_loss‚1.3551689043581006 2loss/SRqqt3/assignment_loss‚1.7368589421468121 1loss/SRqqt4/assignment_loss‚1.180281083124715 1loss/SRqqt2/detection_loss‚3.3927498310804367 epoch‚3 _step‚368 /loss/FRt4/assignment_loss‚2.772857031306705 /loss/FRt3/detection_loss‚3.1787496387958525 1loss/SRqqt3/detection_loss‚2.8141681221979007 trainer/global_step‚4499 % _timestamp‚1.7884779045971005e+09 0loss/FRt1/assignment_loss‚1.8981274096295238 .loss/FRt3/assignment_loss‚6.17097858786583 1loss/FBt4/assignment_loss‚0.28156055905310495 1loss/FBt2/assignment_loss‚0.32226035913711143 /loss/FBt1/detection_loss‚2.8564903487761817 _runtime‚8111.354041992@Í5ˆ^ÿØ 0loss/FBt3/assignment_loss‚0.3812128222327576 .loss/FRt2/detection_loss‚1.822872066176405 .loss/FRt4/detection_loss‚2.714433477537052 1loss/SRqqt1/detection_loss‚1.1064649063252634 2loss/SRqqt1/assignment_loss‚1.3551689043581006 2loss/SRqqt3/assignment_loss‚1.7368589421468121 1loss/SRqqt4/assignment_loss‚1.180281083124715 1loss/SRqqt2/detection_loss‚3.3927498310804367 epoch‚3 _step‚368 /loss/FRt4/assignment_loss‚2.772857031306705 /loss/FRt3/detection_loss‚3.1787496387958525 1loss/SRqqt3/detection_loss‚2.8141681221979007 trainer/global_step‚4499 % _timestamp‚1.7884779045971005e+09 0loss/FRt1/assignment_loss‚1.8981274096295238 .loss/FRt3/assignment_loss‚6.17097858786583 1loss/FBt4/assignment_loss‚0.28156055905310495 1loss/FBt2/assignment_loss‚0.32226035913711143 /loss/FBt1/detection_loss‚2.8564903487761817 _runtime‚8111.354041992 1loss/SRqqt4/detection_loss‚2.1880826821198336 .loss/FBt2/detection_loss‚0.867363193367101 %loss/total_loss‚49.91179714350245 /loss/FRt2/assignment_loss‚3.070923077125175 1loss/FBt1/assignment_loss‚0.46301404712524646 /loss/FRt1/detection_loss‚1.7182779138278215 /loss/FBt3/detection_loss‚2.5803766780429416 2loss/SRqqt2/assignment_loss‚2.2128779341777167 .loss/FBt4/detection_loss‚2.825647527972857ðu‹ÐLƒ€‚(:ø ÔƒèÔÓѲx proc.cpu.threads‚17) disk./.usagePercent‚78.06050888604199% disk./.usageGB‚1163.1579933166504 disk.sda2.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 20611.0546875) disk.nvme0c0n1.out‚194100.01904296875 network.recv‚ 2.589769e+06 disk.dm-1.in‚0 disk.sda3.out‚ 246.13671875 disk.drbd1260.out‚ 1.1328125! disk.nvme0n1.in‚ 20611.0546875) proc.memory.percent‚19.92180347442627$ disk.dm-0.out‚194100.01904296875 disk.dm-1.out‚ 246.13671875 disk.sda.in‚0 disk.sda.out‚ 246.13671875 disk.sda3.in‚0$ memory_percent‚95.66473960876465' proc.memory.availableMB‚ 710.2890625 cpu‚25.16553161570178 network.sent‚ 1.2444197e+07 disk.dm-0.in‚ 20611.0546875 disk.drbd1260.in‚0' disk.nvme0n1.out‚194100.01904296875 disk.sda2.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.98828125’Ȩ’‚(:‡ ÔƒèÔàÒÅx gpu.0.gpu‚71.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚247.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sôaé…‚‚(:ú ãƒèÔ䆤x disk.dm-1.in‚0 disk.sda.out‚ 246.57421875$ memory_percent‚95.66471576690674 network.recv‚ 2.596709e+06 disk.drbd1260.out‚ 1.1328125 disk.sda2.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 20611.0546875 disk.sda.in‚0# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 disk.drbd1260.in‚0 disk.dm-0.in‚ 20611.0546875 disk.dm-1.out‚ 246.57421875( proc.memory.availableMB‚ 710.29296875 proc.cpu.threads‚17! disk.nvme0n1.in‚ 20611.0546875' disk.nvme0n1.out‚194279.47998046875 disk.sda2.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚194279.47998046875$ disk.dm-0.out‚194279.47998046875 cpu‚25.165716750288563 network.sent‚ 1.2553177e+07) disk./.usagePercent‚78.03969137738771% disk./.usageGB‚1162.8477973937988 disk.sda3.in‚0 disk.sda3.out‚ 246.57421875m¾GÈ›ƒ‚(: ãƒèÔ©ø¼x gpu.0.gpu‚9.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚148.156' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.631200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ã8b’„‚(:‡ òƒèÔïñÁw gpu.0.gpu‚94.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚190.214' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.0428 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z!ÑRx…‚(:í òƒèÔžÇÇx disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.in‚0% disk./.usageGB‚1162.8477973937988 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 20747.515625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 247.15625( proc.memory.availableMB‚ 710.04296875# proc.memory.rssMB‚ 3263.98828125' disk.nvme0n1.out‚194530.20947265625 disk.dm-0.in‚ 20747.515625 disk.dm-1.out‚ 247.15625 disk.sda.out‚ 247.15625$ memory_percent‚95.66624164581299 proc.cpu.threads‚17 network.sent‚ 1.2558277e+07 disk.sda2.out‚0) disk.nvme0c0n1.out‚194530.20947265625$ disk.dm-0.out‚194530.20947265625) proc.memory.percent‚19.92180347442627 cpu‚25.165838015160944 network.recv‚ 2.6002e+06) disk./.usagePercent‚78.03969137738771 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.13671875 disk.nvme0n1.in‚ 20747.515625w` ^{†‚(:ð „èÔ‘ÇÅx proc.cpu.threads‚17 network.sent‚ 1.2564411e+07$ disk.dm-0.out‚194875.82275390625 disk.dm-1.out‚ 247.734375' disk.nvme0n1.out‚194875.82275390625 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚95.66471576690674# proc.memory.rssMB‚ 3263.98828125" disk.nvme0c0n1.in‚ 20747.515625 disk.dm-0.in‚ 20747.515625 disk.sda.out‚ 247.734375 disk.sda3.out‚ 247.734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.13671875 disk.nvme0n1.in‚ 20747.515625 disk.sda2.out‚0 network.recv‚ 2.604813e+06% disk./.usageGB‚1162.8244323730469) disk.nvme0c0n1.out‚194875.82275390625 disk.sda.in‚0 disk.sda1.out‚0( disk./.usagePercent‚78.0381233312405 disk.dm-1.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 710.29296875) proc.memory.percent‚19.92180347442627 cpu‚25.16602142215568¯nÇÅ‘‡‚(:† „èÔÓ½³y gpu.0.gpu‚46.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚158.189' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.6378 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Ä8|ˆ‚(:ñ „èÔ† x cpu‚25.166205775050518 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 20747.515625' proc.memory.availableMB‚ 710.2890625# proc.memory.rssMB‚ 3263.98828125 network.recv‚ 2.608054e+06) disk./.usagePercent‚77.96710095306895 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 248.16015625) proc.memory.percent‚19.92180347442627 network.sent‚ 1.256929e+07% disk./.usageGB‚1161.7661476135254 disk.sda3.out‚ 248.16015625 disk.drbd1260.out‚1.140625( disk.nvme0c0n1.out‚195512.5771484375# disk.dm-0.out‚195512.5771484375 disk.sda.out‚ 248.16015625 disk.nvme0n1.in‚ 20747.515625& disk.nvme0n1.out‚195512.5771484375 disk.dm-0.in‚ 20747.515625$ memory_percent‚95.66473960876465[ž«ú‰‚(:… „èÔØÀðy gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.063' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0126 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Ãvb‘Š‚(:† Ÿ„èÔÚÍ«w gpu.0.gpu‚73.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.027 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 cfç}‹‚(:ò Ÿ„èÔ“ÙŽx disk.sda3.in‚0 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 20747.515625( disk.nvme0c0n1.out‚195738.9794921875 disk.dm-0.in‚ 20747.515625 disk.dm-1.in‚0 disk.dm-1.out‚ 248.94140625 disk.drbd1260.out‚1.140625# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 network.sent‚ 1.2575685e+07) disk./.usagePercent‚77.96710095306895 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.66473960876465' proc.memory.availableMB‚ 710.2890625 cpu‚25.166358266931663% disk./.usageGB‚1161.7661476135254& disk.nvme0n1.out‚195738.9794921875# disk.dm-0.out‚195738.9794921875 disk.sda.out‚ 248.94140625 disk.sda3.out‚ 248.94140625 proc.cpu.threads‚17 network.recv‚ 2.613034e+06 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 20747.515625 disk.sda.in‚0ÕF\p2Œ- _wandbruntime‚8203  _runtime‚8203e_ÿÑÇÁ & lr-AdamW/pg2‚0.0005116922112908308 trainer/global_step‚4549 % _timestamp‚1.7884779967899933e+09 _runtime‚8203.547231657 _step‚369 & lr-AdamW/pg1‚0.0005116922112908308ivÈÌŽÆ & lr-AdamW/pg1‚0.0005116922112908308 & lr-AdamW/pg2‚0.0005116922112908308 trainer/global_step‚4549 % _timestamp‚1.7884779967899933e+09 _runtime‚8203.547231657 _step‚369ñ¡1Å:2- _wandbruntime‚8203  _runtime‚8203+kƒ¥? †P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113730249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95880741 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11051507485016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11074514501632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101966550 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60597081 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731149686234624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11055116147700 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41369469(‘‹P YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720078782393856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚667053585002496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚702950817224336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60597076 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720078808785408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚727 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95880741 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11051507485016 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11074514501632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚732 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101966550 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60597081 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731149686234624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11055116147700 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41369469 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113730249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95901159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113781182 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15245076007424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11076924996608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113754816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11057522451194 _step‚371 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11073314322432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209682347 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚714155060253774 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15244999888384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11053913788302 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113781182 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60609886 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41378125 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101987323 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209682347 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113754816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚667196034442752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚703101146465472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209655975 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651951034554368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60609881 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95901165 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚703101171893440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720232559952896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731309511341056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41377442 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 trainer/global_step‚4549 _runtime‚8204.669158194 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11053913788302 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113781182 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60609886 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41378125 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚76119040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101987323 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209682347 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113754816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚667196034442752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚703101146465472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209655975 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651951034554368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60609881 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95901165 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚703101171893440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720232559952896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStat; fsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731309511341056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41377442 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 trainer/global_step‚4549 _runtime‚8204.669158194 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95901159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113781182 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15245076007424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11076924996608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113754816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11057522451194 _step‚371 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11073314322432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209682347 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚714155060253774 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15244999888384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ó•ë|¹2•- _wandbruntime‚8204  _runtime‚8204ß yW–Ñ 0loss/SRqqt3/detection_loss‚2.987125478007577 0loss/FRt4/assignment_loss‚3.5177364174057457 2loss/SRqqt3/assignment_loss‚1.2735080136494203 1loss/FBt2/assignment_loss‚0.15436546027194709 /loss/FRt1/detection_loss‚1.6103288461533587 /loss/FRt4/detection_loss‚2.8885718689245334 1loss/SRqqt1/detection_loss‚1.0702210844383906 # _timestamp‚1.78847799791556e+09 1loss/FBt4/assignment_loss‚0.17287872600051438 2loss/SRqqt4/assignment_loss‚1.7699846821395975 .loss/FBt4/detection_loss‚2.274034731917911 trainer/global_step‚4549 0loss/FRt1/assignment_loss‚2.3064406541260807 /loss/FBt3/assignment_loss‚0.570385321657384 /loss/FBt1/detection_loss‚3.0037852355412076 -loss/FBt3/detection_loss‚2.13680343684696 0loss/FRt2/assignment_loss‚2.6233836868976024 1loss/SRqqt2/assignment_loss‚2.651866976420085 .loss/FRt2/detection_loss‚1.973445880127714 .loss/FRt3/detection_loss‚3.647259102927314 0loss/FBt1/assignment_loss‚0.1072821147473795 _step‚372 /loss/FRt3/assignment_loss‚4.404077794816759 2loss/SRqqt1/assignment_loss‚1.1302773110347533 1loss/SRqqt2/detection_loss‚3.2957156817118327 1loss/SRqqt4/detection_loss‚2.0742634009373817 /loss/FBt2/detection_loss‚0.9339816073576609 &loss/total_loss‚48.577723514059116 epoch‚3 _runtime‚8204.670937628‹ÜÇ7\—Ö &loss/total_loss‚48.577723514059116 epoch‚3 _runtime‚8204.670937628 0loss/SRqqt3/detection_loss‚2.987125478007577 0loss/FRt4/assignment_loss‚3.5177364174057457 2loss/SRqqt3/assignment_loss‚1.2735080136494203 1loss/FBt2/assignment_loss‚0.15436546027194709 /loss/FRt1/detection_loss‚1.6103288461533587 /loss/FRt4/detection_loss‚2.8885718689245334 1loss/SRqqt1/detection_loss‚1.0702210844383906 # _timestamp‚1.78847799791556e+09 1loss/FBt4/assignment_loss‚0.17287872600051438 2loss/SRqqt4/assignment_loss‚1.7699846821395975 .loss/FBt4/detection_loss‚2.274034731917911 trainer/global_step‚4549 0loss/FRt1/assignment_loss‚2.3064406541260807 /loss/FBt3/assignment_loss‚0.570385321657384 /loss/FBt1/detection_loss‚3.0037852355412076 -loss/FBt3/detection_loss‚2.13680343684696 0loss/FRt2/assignment_loss‚2.6233836868976024 1loss/SRqqt2/assignment_loss‚2.651866976420085 .loss/FRt2/detection_loss‚1.973445880127714 .loss/FRt3/detection_loss‚3.647259102927314 0loss/FBt1/assignment_loss‚0.1072821147473795 _step‚372 /loss/FRt3/assignment_loss‚4.404077794816759 2loss/SRqqt1/assignment_loss‚1.1302773110347533 1loss/SRqqt2/detection_loss‚3.2957156817118327 1loss/SRqqt4/detection_loss‚2.0742634009373817 /loss/FBt2/detection_loss‚0.9339816073576609ôI±ñ2ƒ˜‚(:ø ®„èÔ뀦y disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 710.0078125 proc.cpu.threads‚17' disk.nvme0n1.out‚196088.12158203125 disk.sda2.out‚0) disk.nvme0c0n1.out‚196088.12158203125 disk.dm-0.in‚20771.38671875 disk.dm-1.in‚0 disk.sda.out‚ 249.3203125 disk.sda3.out‚ 249.3203125$ memory_percent‚95.66645622253418) disk./.usagePercent‚77.96710607321963 disk.sda2.in‚0 disk.sda1.in‚0$ disk.dm-0.out‚196088.12158203125 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 cpu‚25.16650187026066 network.sent‚ 1.258039e+07% disk./.usageGB‚1161.7662239074707" disk.nvme0n1.in‚20771.38671875$ disk.nvme0c0n1.in‚20771.38671875 disk.dm-1.out‚ 249.3203125 disk.drbd1260.out‚ 1.14453125 network.recv‚ 2.615681e+06«¯T‘™‚(:† ®„èÔâ÷¼{ gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚178.264' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 À|‚š‚(:÷ ½„èÔþ“©x) disk.nvme0c0n1.out‚196339.84130859375 disk.dm-1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 710.1328125# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17 network.recv‚ 2.622369e+06) disk./.usagePercent‚77.96710760926483 disk.dm-0.in‚ 20878.7890625 disk.sda3.out‚ 249.7421875 disk.sda1.out‚0$ memory_percent‚95.66569328308105 network.sent‚ 1.2688333e+07# disk.nvme0c0n1.in‚ 20878.7890625$ disk.dm-0.out‚196339.84130859375 disk.dm-1.out‚ 249.7421875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda.out‚ 249.7421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.14453125! disk.nvme0n1.in‚ 20878.7890625' disk.nvme0n1.out‚196339.84130859375 disk.sda2.in‚0 cpu‚25.166688767406857% disk./.usageGB‚1161.7662467956543JÂM ’›‚(:‡ ½„èÔæîÒx gpu.0.gpu‚54.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚203.498' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.6996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“©-Cœ‚(:… Ì„èÔÔî‘y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.127' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð Ø3~‚(:ó Ì„èÔßÐy disk.sda.in‚0 disk.sda.out‚ 250.0859375# disk.nvme0c0n1.in‚ 20878.7890625( disk.nvme0c0n1.out‚196457.2216796875 network.sent‚ 1.2693216e+07% disk./.usageGB‚1161.3321266174316 disk.dm-0.in‚ 20878.7890625) proc.memory.percent‚19.92180347442627 cpu‚25.166860564473208 proc.cpu.threads‚17) disk./.usagePercent‚77.93797343985322# disk.dm-0.out‚196457.2216796875 disk.dm-1.in‚0& disk.nvme0n1.out‚196457.2216796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3263.98828125 disk.sda3.in‚0 disk.sda3.out‚ 250.0859375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.1484375! disk.nvme0n1.in‚ 20878.7890625 disk.sda1.out‚0$ memory_percent‚95.66569328308105' proc.memory.availableMB‚ 710.1328125 network.recv‚ 2.625947e+06 disk.dm-1.out‚ 250.0859375F=‹‚ž‚(:÷ Û„èÔ³þy proc.cpu.threads‚17 network.recv‚ 2.630674e+06 disk.dm-1.in‚0 disk.sda3.out‚ 250.45703125 disk.drbd1260.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627% disk./.usageGB‚1160.6809272766113 disk.dm-0.in‚ 20878.7890625# disk.dm-0.out‚196781.0302734375 disk.sda.in‚0! disk.nvme0n1.in‚ 20878.7890625& disk.nvme0n1.out‚196781.0302734375 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 1.2699547e+07) disk./.usagePercent‚77.89427090569812 disk.drbd1260.out‚ 1.1484375$ memory_percent‚95.66633701324463( proc.memory.availableMB‚ 710.02734375 cpu‚25.166798156529758 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 20878.7890625( disk.nvme0c0n1.out‚196781.0302734375 disk.dm-1.out‚ 250.45703125 disk.sda.out‚ 250.45703125 disk.sda1.in‚0°ã9 ’Ÿ‚(:‡ Û„èÔèâÅz gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚165.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04Zº™’ ‚(:‡ ê„èԾϑx gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚292.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êàèD‚¡‚(:÷ ê„èÔæêÂz! disk.nvme0n1.in‚ 20878.7890625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 20878.7890625 disk.dm-1.in‚0 disk.sda3.out‚ 250.94140625$ memory_percent‚95.66638469696045 cpu‚25.16664508050926 proc.cpu.threads‚17 network.sent‚ 1.2704591e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.15234375# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627( disk.nvme0c0n1.out‚197168.4619140625 disk.sda.out‚ 250.94140625 disk.sda3.in‚0( proc.memory.availableMB‚ 710.01953125 network.recv‚ 2.634053e+06) disk./.usagePercent‚77.89424120882416% disk./.usageGB‚1160.6804847717285& disk.nvme0n1.out‚197168.4619140625 disk.sda1.in‚0 disk.dm-0.in‚ 20878.7890625# disk.dm-0.out‚197168.4619140625 disk.dm-1.out‚ 250.941406257X÷2¢- _wandbruntime‚8274  _runtime‚8274Èp"Ç£Á & lr-AdamW/pg1‚0.0005078069399956056 & lr-AdamW/pg2‚0.0005078069399956056 trainer/global_step‚4599 % _timestamp‚1.7884780673754873e+09 _runtime‚8274.131248143 _step‚373|ƒÌ¤Æ & lr-AdamW/pg2‚0.0005078069399956056 trainer/global_step‚4599 % _timestamp‚1.7884780673754873e+09 _runtime‚8274.131248143 _step‚373 & lr-AdamW/pg1‚0.0005078069399956056õÌ™kí2¥- _wandbruntime‚8274  _runtime‚8274·È"4(¦ˆP F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211863886 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚738962739736064 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103046419 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96901935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚721642749612312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41809093 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚658776263614976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884780673765182e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114961951 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚738959102683648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721639115521660 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114988291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚727767660345856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114961951 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61237326 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚727767686737408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96901935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61237331 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚674183775227392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96901941 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚674183902562816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15407587744768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15407511612416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11195052998656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚727767660345856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11175444903208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11191442337792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41809793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114988291 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211890232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚710467279281136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103047124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚738959102683648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11191442337792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187ƒ6ö½Æ§P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114988291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚727767660345856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114961951 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61237326 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚727767686737408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96901935 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61237331 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚674183775227392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96901941 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚674183902562816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15407587744768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15407511612416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11195052998656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚727767660345856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11175444903208 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11191442337792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41809793 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚76132352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114988291 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211890232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚710467279281136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103047124 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚705 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚738959102683648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11191442337792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211863886 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚738962739736064 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103046419 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96901935 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72164ìŠ|›M2749612312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41809093 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚658776263614976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884780673765182e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114961951 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚738959102683648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721639115521660öžfG2¨- _wandbruntime‚8275  _runtime‚8275‡4pÈ'©N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚710617633950240 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚721791851066082 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚727921437904896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚727921464296448 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11193852819456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚674326374127104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15410820123136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96922365 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚658915605207040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚658915554003968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96922365 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96922359 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114986518 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚674326499352064 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103068052 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96922359 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114986518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115012884 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211935249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚710617608522272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚727921464296448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211908877 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115012884 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61250131 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41817921 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚721795485156942 _step‚375 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11177851206702 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15410894145024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211908877 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11197463493632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103068736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 $ _timestamp‚1.788478068435508e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11197463493632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚727921437904896 _runtime‚ 8275.19161212 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61250136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚111742425438104—»¯'ª•N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96922365 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96922359 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114986518 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚674326499352064 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103068052 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96922359 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114986518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115012884 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211935249 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚710617608522272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚727921464296448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211908877 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115012884 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61250131 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41817921 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚721795485156942 _step‚375 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11177851206702 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15410894145024 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211908877 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11197463493632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103068736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 $ _timestamp‚1.788478068435508e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11197463493632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚727921437904896 _runtime‚ 8275.19161212 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61250136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11174242543810 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚710617633950240 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚721791851066082 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚727921437904896 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚727921464296448 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11193852819456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚674326374127104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15410820123136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96922365 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚658915605207040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚658915554003968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892÷êÞË2«- _wandbruntime‚8275  _runtime‚8275úΔ^¬Ø 1loss/SRqqt1/detection_loss‚0.9543647204856734 /loss/FBt2/detection_loss‚1.1559617900504515 _runtime‚8275.193775667 /loss/FRt4/detection_loss‚2.7939653415232897 /loss/FBt4/detection_loss‚2.7416883273558184 0loss/FRt2/assignment_loss‚2.4174670070972084 2loss/SRqqt1/assignment_loss‚1.1944975716885455 0loss/FBt3/assignment_loss‚0.2548447593126912 /loss/FRt2/detection_loss‚1.8322976560876616 1loss/FBt1/assignment_loss‚0.41456581612909427 2loss/SRqqt3/assignment_loss‚1.5844236517494374 1loss/FBt4/assignment_loss‚0.28082412178628147 0loss/SRqqt4/detection_loss‚2.333040281198919 % _timestamp‚1.7884780684383886e+09 _step‚376 0loss/FRt1/assignment_loss‚1.8382082374559507 /loss/FRt3/assignment_loss‚4.860351499758269 2loss/SRqqt2/assignment_loss‚1.8724287245422602 2loss/SRqqt4/assignment_loss‚2.2085993951186538 /loss/FRt1/detection_loss‚1.5350612932832317 epoch‚3 trainer/global_step‚4599 /loss/FRt4/assignment_loss‚3.888343144208193 /loss/FRt3/detection_loss‚3.4318928655825163 0loss/SRqqt3/detection_loss‚2.753483620556918 /loss/FBt1/detection_loss‚1.9793669093738904 /loss/FBt3/detection_loss‚2.1335216474533083 %loss/total_loss‚46.85764860112379 1loss/FBt2/assignment_loss‚0.24002658197512994 1loss/SRqqt2/detection_loss‚2.1584236373504004sgîc­Ý /loss/FRt4/detection_loss‚2.7939653415232897 /loss/FBt4/detection_loss‚2.7416883273558184 0loss/FRt2/assignment_loss‚2.4174670070972084 2loss/SRqqt1/assignment_loss‚1.1944975716885455 0loss/FBt3/assignment_loss‚0.2548447593126912 /loss/FRt2/detection_loss‚1.8322976560876616 1loss/FBt1/assignment_loss‚0.41456581612909427 2loss/SRqqt3/assignment_loss‚1.5844236517494374 1loss/FBt4/assignment_loss‚0.28082412178628147 0loss/SRqqt4/detection_loss‚2.333040281198919 % _timestamp‚1.7884780684383886e+09 _step‚376 0loss/FRt1/assignment_loss‚1.8382082374559507 /loss/FRt3/assignment_loss‚4.860351499758269 2loss/SRqqt2/assignment_loss‚1.8724287245422602 2loss/SRqqt4/assignment_loss‚2.2085993951186538 /loss/FRt1/detection_loss‚1.5350612932832317 epoch‚3 trainer/global_step‚4599 /loss/FRt4/assignment_loss‚3.888343144208193 /loss/FRt3/detection_loss‚3.4318928655825163 0loss/SRqqt3/detection_loss‚2.753483620556918 /loss/FBt1/detection_loss‚1.9793669093738904 /loss/FBt3/detection_loss‚2.1335216474533083 %loss/total_loss‚46.85764860112379 1loss/FBt2/assignment_loss‚0.24002658197512994 1loss/SRqqt2/detection_loss‚2.1584236373504004 1loss/SRqqt1/detection_loss‚0.9543647204856734 /loss/FBt2/detection_loss‚1.1559617900504515 _runtime‚8275.193775667øßH’®‚(:‡ ù„èÔÁôö{ gpu.0.gpu‚83.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚116.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mµÆƒ¯‚(:ø ù„èÔ˜á| disk.sda3.in‚0( proc.memory.availableMB‚ 709.85546875) proc.memory.percent‚19.92180347442627 cpu‚25.166541224153246 proc.cpu.threads‚17$ disk./.usageGB‚1160.517490386963 disk.sda1.out‚0# disk.dm-0.out‚197338.3955078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.15234375# proc.memory.rssMB‚ 3263.98828125 network.sent‚ 1.272376e+07) disk./.usagePercent‚77.88330251890115" disk.nvme0n1.in‚21052.40234375 disk.sda1.in‚0 disk.dm-0.in‚21052.40234375 disk.sda3.out‚ 251.36328125$ memory_percent‚95.66738605499268 network.recv‚ 2.63867e+06& disk.nvme0n1.out‚197338.3955078125 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚197338.3955078125 disk.dm-1.out‚ 251.36328125 disk.sda.in‚0 disk.sda.out‚ 251.36328125$ disk.nvme0c0n1.in‚21052.40234375 disk.dm-1.in‚0ֈ̰‚(:’ ˆ…èÔ•ööw gpu.0.gpu‚80.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚279.791' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üƒ=Þ±‚(:ö ˆ…èÔ¯åx' proc.memory.availableMB‚ 709.9609375 cpu‚25.166357874493137 proc.cpu.threads‚17 disk.sda1.out‚0) disk.nvme0c0n1.out‚197604.58447265625$ disk.dm-0.out‚197604.58447265625 network.sent‚ 1.2818231e+07 network.recv‚ 2.643772e+06" disk.nvme0n1.in‚21052.40234375' disk.nvme0n1.out‚197604.58447265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 251.6953125 disk.drbd1260.in‚0# memory_percent‚95.6667423248291# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 disk.sda1.in‚0 disk.dm-0.in‚21052.40234375) disk./.usagePercent‚77.88478659457712% disk./.usageGB‚1160.5396041870117 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 251.6953125 disk.sda3.in‚0$ disk.nvme0c0n1.in‚21052.40234375 disk.dm-1.out‚ 251.6953125 disk.drbd1260.out‚1.15625Á’”‚²‚(:÷ —…èÔ÷ªy disk.sda1.in‚0$ disk.nvme0c0n1.in‚21052.40234375 disk.dm-0.in‚21052.40234375# memory_percent‚95.6667423248291# proc.memory.rssMB‚ 3263.98828125 network.sent‚ 1.2824625e+07 disk.drbd1260.out‚1.15625" disk.nvme0n1.in‚21052.40234375' disk.nvme0n1.out‚197811.81884765625 disk.sda1.out‚0) disk.nvme0c0n1.out‚197811.81884765625' proc.memory.availableMB‚ 709.9609375) disk./.usagePercent‚77.89519432487414 disk.sda.out‚ 252.44140625) proc.memory.percent‚19.92180347442627 cpu‚25.16656335644463 proc.cpu.threads‚17 network.recv‚ 2.64871e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 252.44140625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0$ disk.dm-0.out‚197811.81884765625% disk./.usageGB‚1160.6946868896484 disk.sda3.out‚ 252.44140625 disk.sda2.out‚0¼³Ÿ‘³‚(:† —…èÔÏÚÀz gpu.0.gpu‚38.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚112.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5696 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶Þ ñœ´‚(:‘ ¦…èÔÈô™w gpu.0.gpu‚39.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚278.938' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.78759999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÜ=w„µ‚(:ù ¦…èÔ‘ÀÏy) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚21052.40234375( disk.nvme0c0n1.out‚197910.8095703125 disk.sda1.out‚0( proc.memory.availableMB‚ 709.95703125 cpu‚25.166621008562423 network.recv‚ 2.651711e+06 disk.dm-1.out‚ 252.91015625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚95.66676616668701( disk./.usagePercent‚77.8973652687645$ disk./.usageGB‚1160.727035522461 disk.dm-0.in‚21052.40234375# disk.dm-0.out‚197910.8095703125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 252.91015625 disk.sda3.out‚ 252.91015625 network.sent‚ 1.2829626e+07 disk.drbd1260.out‚ 1.16015625" disk.nvme0n1.in‚21052.40234375& disk.nvme0n1.out‚197910.8095703125# proc.memory.rssMB‚ 3263.98828125€t¼’¶‚(:‡ µ…èÔÒÙêx gpu.0.gpu‚68.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚277.389' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S·fr·‚(:ç µ…èÔõš¢{ network.sent‚ 1.2835647e+07) disk./.usagePercent‚77.90943192788338( disk.nvme0c0n1.out‚198342.9267578125 disk.dm-1.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 21176.796875 disk.sda1.out‚0$ proc.memory.availableMB‚709.8125 network.recv‚ 2.656277e+06$ disk./.usageGB‚1160.906837463379" disk.nvme0c0n1.in‚ 21176.796875 disk.dm-0.in‚ 21176.796875# disk.dm-0.out‚198342.9267578125 disk.sda.out‚ 253.21875 disk.sda3.out‚ 253.21875 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 cpu‚25.166492418573576 disk.dm-1.out‚ 253.21875 disk.drbd1260.out‚ 1.16015625& disk.nvme0n1.out‚198342.9267578125$ memory_percent‚95.66764831542969 proc.cpu.threads‚17tJ¡2¸- _wandbruntime‚8341  _runtime‚8341¸/U€Æ¹À & lr-AdamW/pg2‚0.0005038837649954397 trainer/global_step‚4649 $ _timestamp‚1.788478134951208e+09 _runtime‚8341.707527604 _step‚377 & lr-AdamW/pg1‚0.0005038837649954397î7¡ŸËºÅ $ _timestamp‚1.788478134951208e+09 _runtime‚8341.707527604 _step‚377 & lr-AdamW/pg1‚0.0005038837649954397 & lr-AdamW/pg2‚0.0005038837649954397 trainer/global_step‚4649ù=8\2»- _wandbruntime‚8341  _runtime‚8341`K¤öd¼ˆP F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214116788 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116219993 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚735456564689408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 A8DeviceStatsMonitor.on_train_batch_start/num_devicº^ª%e_alloc‚3719 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11311980834816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717983741337936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚746768519132672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104127174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚735456564689408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61877581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42249598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚735456538297856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214143134 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15573604487680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11315591495680 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104127876 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11295773658716 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97923141 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11311980834816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚746768519132672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116193653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681314512790528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729275906333968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97923141 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97923135 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚717983766765904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448)>1(½P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11311980834816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717983741337936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚746768519132672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104127174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚735456564689408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61877581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42249598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚735456538297856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214143134 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15573604487680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11315591495680 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104127876 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11295773658716 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97923141 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11311980834816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚746768519132672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116193653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681314512790528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729275906333968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97923141 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97923135 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚717983766765904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214116788 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116219993 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚735456564689408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921úì¸3µ2¾- _wandbruntime‚8342  _runtime‚8342‘DÕi'¿‘N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61890381 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 _step‚379 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚735610342248448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚746928344239104 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11294571299318 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97943559 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97943559 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11318001990656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚665880124656640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97943565 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11314391316480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11318001990656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214161779 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚735610342248448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97943565 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4649 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718134070579072 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚8342.903495817 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104148734 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15576941044736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214188151 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104149397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚665880073453568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11298179962210 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214161779 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116218220 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚735610315856896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116244586 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187DïÁ_× À–N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11318001990656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚665880124656640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97943565 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11314391316480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11318001990656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214161779 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚735610342248448 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97943565 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4649 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718134070579072 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚8342.903495817 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104148734 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15576941044736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214188151 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104149397 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚665880073453568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11298179962210 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214161779 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116218220 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚735610315856896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116244586 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61890381 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 _step‚379 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚735610342248448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚746928344239104 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11294571299318 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97943559 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97943559 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424ûø cs2Á- _wandbruntime‚8342  _runtime‚8342a2YÓXÂÒ 2loss/SRqqt1/assignment_loss‚1.2021106992054869 1loss/FBt4/assignment_loss‚0.25294896643882725 trainer/global_step‚4649 /loss/FRt4/assignment_loss‚4.088161736726761 0loss/SRqqt2/detection_loss‚2.873444080352783 epoch‚3 _runtime‚8342.905011884 _step‚380 2loss/SRqqt3/assignment_loss‚1.8839502784389037 0loss/FBt2/assignment_loss‚0.1219697555761262 % _timestamp‚1.7884781361484325e+09 0loss/FRt1/assignment_loss‚1.9005292684867465 0loss/SRqqt3/detection_loss‚2.514394583525481 .loss/FBt3/detection_loss‚2.511931899935007 2loss/SRqqt4/assignment_loss‚1.8957163695155121 /loss/FRt1/detection_loss‚1.7412281122600972 1loss/SRqqt1/detection_loss‚1.1517361296444304 %loss/total_loss‚47.99109625917336 /loss/FRt3/assignment_loss‚4.038760238223606 0loss/FBt3/assignment_loss‚0.3168453675834826 0loss/FRt2/assignment_loss‚2.7517883193259145 1loss/FBt1/assignment_loss‚0.15569400072046038 .loss/FRt3/detection_loss‚3.342532065179613 .loss/FRt4/detection_loss‚3.163032777607441 /loss/FBt1/detection_loss‚2.5311630103323193 /loss/FBt2/detection_loss‚0.8969046957913394 .loss/FBt4/detection_loss‚2.538802914759692 2loss/SRqqt2/assignment_loss‚1.9051568790485984 .loss/FRt2/detection_loss‚2.002298336987402 1loss/SRqqt4/detection_loss‚2.2099957735073277ùAm]Ã× 2loss/SRqqt1/assignment_loss‚1.2021106992054869 1loss/FBt4/assignment_loss‚0.25294896643882725 trainer/global_step‚4649 /loss/FRt4/assignment_loss‚4.088161736726761 0loss/SRqqt2/detection_loss‚2.873444080352783 epoch‚3 _runtime‚8342.905011884 _step‚380 2loss/SRqqt3/assignment_loss‚1.8839502784389037 0loss/FBt2/assignment_loss‚0.1219697555761262 % _timestamp‚1.7884781361484325e+09 0loss/FRt1/assignment_loss‚1.9005292684867465 0loss/SRqqt3/detection_loss‚2.514394583525481 .loss/FBt3/detection_loss‚2.511931899935007 2loss/SRqqt4/assignment_loss‚1.8957163695155121 /loss/FRt1/detection_loss‚1.7412281122600972 1loss/SRqqt1/detection_loss‚1.1517361296444304 %loss/total_loss‚47.99109625917336 /loss/FRt3/assignment_loss‚4.038760238223606 0loss/FBt3/assignment_loss‚0.3168453675834826 0loss/FRt2/assignment_loss‚2.7517883193259145 1loss/FBt1/assignment_loss‚0.15569400072046038 .loss/FRt3/detection_loss‚3.342532065179613 .loss/FRt4/detection_loss‚3.163032777607441 /loss/FBt1/detection_loss‚2.5311630103323193 /loss/FBt2/detection_loss‚0.8969046957913394 .loss/FBt4/detection_loss‚2.538802914759692 2loss/SRqqt2/assignment_loss‚1.9051568790485984 .loss/FRt2/detection_loss‚2.002298336987402 1loss/SRqqt4/detection_loss‚2.2099957735073277üåâexÄ‚(:… Ä…èÔðÛÙy gpu.0.gpu‚62.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁÙxÅ‚(:í Ä…èÔÌ¢òy disk.sda1.in‚0( proc.memory.availableMB‚ 709.75390625) disk./.usagePercent‚77.90953996306283" disk.dm-0.out‚198689.658203125 disk.sda.in‚0 disk.sda.out‚ 253.7421875 disk.sda2.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚1160.908447265625' disk.nvme0c0n1.out‚198689.658203125 disk.dm-0.in‚ 21176.796875 disk.dm-1.in‚0 disk.dm-1.out‚ 253.7421875 disk.sda3.in‚0 disk.nvme0n1.in‚ 21176.796875% disk.nvme0n1.out‚198689.658203125 network.sent‚ 1.2840644e+07 network.recv‚ 2.659739e+06" disk.nvme0c0n1.in‚ 21176.796875 disk.drbd1260.out‚ 1.1640625$ memory_percent‚95.66800594329834# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 cpu‚25.166649655942088 disk.sda3.out‚ 253.7421875 disk.drbd1260.in‚0 disk.sda2.in‚0äP¢xÆ‚(:í Ó…èÔå÷ºw) proc.memory.percent‚19.92180347442627( disk./.usagePercent‚77.9079637246745 disk.dm-1.in‚0 disk.dm-1.out‚ 254.3046875 disk.nvme0n1.in‚ 21176.796875( proc.memory.availableMB‚ 709.50390625# proc.memory.rssMB‚ 3263.98828125 proc.cpu.threads‚17' disk.nvme0c0n1.out‚199050.458984375" disk.dm-0.out‚199050.458984375 disk.sda.out‚ 254.3046875 disk.sda3.out‚ 254.3046875 disk.drbd1260.out‚ 1.1640625% disk.nvme0n1.out‚199050.458984375 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.2949535e+07 network.recv‚ 2.666421e+06% disk./.usageGB‚1160.8849601745605" disk.nvme0c0n1.in‚ 21176.796875 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0 cpu‚25.166782293824657 disk.dm-0.in‚ 21176.796875 disk.sda3.in‚0 disk.sda1.in‚0$ memory_percent‚95.66953182220459pÑîW’Ç‚(:‡ Ó…èÔ±ùùw gpu.0.gpu‚74.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.519' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1038 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›¡ßyÈ‚(:î â…èÔ£§Ž{ disk.sda2.out‚0" proc.memory.availableMB‚709.75) disk./.usagePercent‚77.91037582766211% disk./.usageGB‚1160.9209022521973" disk.nvme0c0n1.in‚ 21176.796875) disk.nvme0c0n1.out‚199209.15869140625' disk.nvme0n1.out‚199209.15869140625 disk.sda1.out‚0$ memory_percent‚95.66802978515625 cpu‚25.166913594171877 network.sent‚ 1.2954408e+07 network.recv‚ 2.66949e+06 disk.sda.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3263.98828125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 254.6796875 disk.sda.out‚ 254.6796875 disk.sda3.out‚ 254.6796875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.16796875 disk.sda2.in‚0) proc.memory.percent‚19.92180347442627 disk.dm-0.in‚ 21176.796875$ disk.dm-0.out‚199209.15869140625 disk.sda3.in‚0 disk.nvme0n1.in‚ 21176.796875î‘ɂ(:† â…èÔ‡¬{ gpu.0.gpu‚26.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.047' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.0094 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…¯’Ê‚(:‡ ñ…èÔ¢†éw gpu.0.gpu‚65.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.9574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó_±L€Ë‚(:õ ñ…èÔ“Ý™x" proc.memory.availableMB‚709.75# proc.memory.rssMB‚ 3263.98828125 network.sent‚ 1.2960525e+07 disk.sda2.in‚0) disk.nvme0c0n1.out‚199209.38916015625 disk.dm-0.in‚ 21312.2734375 cpu‚25.167071574044826 network.recv‚ 2.674074e+06! disk.nvme0n1.in‚ 21312.2734375' disk.nvme0n1.out‚199209.38916015625 disk.sda.in‚0$ memory_percent‚95.66802978515625) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17) disk./.usagePercent‚77.91037582766211% disk./.usageGB‚1160.9209022521973# disk.nvme0c0n1.in‚ 21312.2734375$ disk.dm-0.out‚199209.38916015625 disk.dm-1.in‚0 disk.dm-1.out‚ 255.25390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.16796875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 255.25390625 disk.sda3.in‚0 disk.sda3.out‚ 255.25390625íh›{Ì‚(:𠀆èÔ»Þx( disk.nvme0c0n1.out‚199560.7998046875 network.sent‚ 1.29654e+07 disk.dm-1.out‚ 255.6015625 disk.sda.out‚ 255.6015625 disk.sda3.in‚0& disk.nvme0n1.out‚199560.7998046875( proc.memory.availableMB‚ 709.75390625# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 network.recv‚ 2.677053e+06 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 21312.2734375 disk.sda2.out‚0 disk.dm-0.in‚ 21312.2734375$ memory_percent‚95.66800594329834 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda3.out‚ 255.6015625 disk.sda1.out‚0# disk.dm-0.out‚199560.7998046875 cpu‚25.167218470187862( disk./.usagePercent‚77.9147758291531% disk./.usageGB‚1160.9864654541016 disk.sda.in‚0 disk.drbd1260.out‚1.171875 disk.sda2.in‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 21312.2734375Éf˜q’Í‚(:‡ €†èÔÕ¾áx gpu.0.gpu‚97.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚139.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.8238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Mçv΂(:’ †èÔåÌÏx gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚184.211' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.842200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀIèÏ‚(:ö †èÔû‰µy disk.sda1.out‚0$ memory_percent‚95.66800594329834# proc.memory.rssMB‚ 3263.98828125 disk.dm-1.in‚0 disk.sda3.out‚ 256.01953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.percent‚19.92180347442627 cpu‚25.167414457552955 proc.cpu.threads‚17 network.sent‚ 1.2971796e+07% disk./.usageGB‚1161.1739044189453 disk.dm-1.out‚ 256.01953125 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.171875! disk.nvme0n1.in‚ 21312.2734375 network.recv‚ 2.681943e+06) disk./.usagePercent‚77.92735501535553 disk.dm-0.in‚ 21312.2734375# disk.dm-0.out‚200003.5029296875& disk.nvme0n1.out‚200003.5029296875# disk.nvme0c0n1.in‚ 21312.2734375( disk.nvme0c0n1.out‚200003.5029296875( proc.memory.availableMB‚ 709.75390625 disk.sda.out‚ 256.01953125,þôO2Ð- _wandbruntime‚8434  _runtime‚8434‹ºËÅÑ¿ % lr-AdamW/pg2‚0.000499923503874123 trainer/global_step‚4699 % _timestamp‚1.7884782279697318e+09 _runtime‚8434.728770061 _step‚381 % lr-AdamW/pg1‚0.000499923503874123DE¸ÊÒÄ % lr-AdamW/pg1‚0.000499923503874123 % lr-AdamW/pg2‚0.000499923503874123 trainer/global_step‚4699 % _timestamp‚1.7884782279697318e+09 _runtime‚8434.728770061 _step‚381ýX‰m2Ó- _wandbruntime‚8434  _runtime‚8434h€>t(ÔˆP H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42690047 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚725500203394736 _step‚382 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚754577935581696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚743145416249856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688445247058432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚754577935581696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚13ÓÊ32187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚743145416249856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688445247058432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42690047 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚725500203394736 _step‚382 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448þdþ2Ö- _wandbruntime‚8436  _runtime‚8436p¡GG'×’N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98964759 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15743208784896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216441053 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105228695 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚737065432690698 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚743299220200448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216414681 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚672844644106240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚743299220200448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117449922 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672844592903168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 _runtime‚8436.949370467 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98964765 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42698064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105229380 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15743134763008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98964765 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688587727666176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11414900054826 _step‚383 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚725650532635872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11434929813504 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚754737760688128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚754737760688128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚737069066781558 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62530636 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42698744 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11438540487680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216414681 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117476288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11418508717718 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117476288 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98964759 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0o4Bu'Ø—N @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 _runtime‚8436.949370467 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98964765 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42698064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105229380 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15743134763008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98964765 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688587727666176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11414900054826 _step‚383 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚725650532635872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11434929813504 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚754737760688128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚754737760688128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚737069066781558 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62530636 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42698744 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11438540487680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216414681 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117476288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11418508717718 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117476288 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98964759 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98964759 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15743208784896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216441053 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105228695 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚737065432690698 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚743299220200448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216414681 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚672844644106240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚743299220200448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117449922 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672844592903168ÿ«‹E2Ù- _wandbruntime‚8436  _runtime‚8436C•u“ZÚÔ /loss/FRt4/detection_loss‚3.5435863506226313 /loss/FBt1/detection_loss‚2.2096018408026015 % _timestamp‚1.7884782301968832e+09 2loss/SRqqt1/assignment_loss‚1.1278578146635223 0loss/SRqqt3/detection_loss‚2.600757920742035 .loss/FBt4/detection_loss‚3.231919080018997 /loss/FRt4/assignment_loss‚2.462986560094924 /loss/FRt2/assignment_loss‚2.818064208464189 1loss/FBt4/assignment_loss‚0.14126951443722646 1loss/SRqqt3/assignment_loss‚2.203067898750305 2loss/SRqqt4/assignment_loss‚1.3604401073728998 1loss/FBt1/assignment_loss‚0.05583239042399717 0loss/SRqqt4/detection_loss‚2.186767508586248 $loss/total_loss‚45.1587790929847 epoch‚3 trainer/global_step‚4699 0loss/FRt1/assignment_loss‚1.7506506878938248 /loss/FRt3/assignment_loss‚3.742828475801568 0loss/FBt3/assignment_loss‚0.2611010383417514 /loss/FRt3/detection_loss‚3.3995099381396643 1loss/SRqqt1/detection_loss‚1.0650451028320407 _runtime‚8436.953211717 /loss/FBt2/detection_loss‚0.9711941272131542 /loss/FBt3/detection_loss‚2.3396762822355543 2loss/SRqqt2/assignment_loss‚1.6089858694208994 1loss/SRqqt2/detection_loss‚2.6137234138117895 _step‚384 1loss/FBt2/assignment_loss‚0.05958900204805376 /loss/FRt1/detection_loss‚1.6759433450703787 /loss/FRt2/detection_loss‚1.7283806151964447|}‰_ÛÙ 1loss/FBt2/assignment_loss‚0.05958900204805376 /loss/FRt1/detection_loss‚1.6759433450703787 /loss/FRt2/detection_loss‚1.7283806151964447 /loss/FRt4/detection_loss‚3.5435863506226313 /loss/FBt1/detection_loss‚2.2096018408026015 % _timestamp‚1.7884782301968832e+09 2loss/SRqqt1/assignment_loss‚1.1278578146635223 0loss/SRqqt3/detection_loss‚2.600757920742035 .loss/FBt4/detection_loss‚3.231919080018997 /loss/FRt4/assignment_loss‚2.462986560094924 /loss/FRt2/assignment_loss‚2.818064208464189 1loss/FBt4/assignment_loss‚0.14126951443722646 1loss/SRqqt3/assignment_loss‚2.203067898750305 2loss/SRqqt4/assignment_loss‚1.3604401073728998 1loss/FBt1/assignment_loss‚0.05583239042399717 0loss/SRqqt4/detection_loss‚2.186767508586248 $loss/total_loss‚45.1587790929847 epoch‚3 trainer/global_step‚4699 0loss/FRt1/assignment_loss‚1.7506506878938248 /loss/FRt3/assignment_loss‚3.742828475801568 0loss/FBt3/assignment_loss‚0.2611010383417514 /loss/FRt3/detection_loss‚3.3995099381396643 1loss/SRqqt1/detection_loss‚1.0650451028320407 _runtime‚8436.953211717 /loss/FBt2/detection_loss‚0.9711941272131542 /loss/FBt3/detection_loss‚2.3396762822355543 2loss/SRqqt2/assignment_loss‚1.6089858694208994 1loss/SRqqt2/detection_loss‚2.6137234138117895 _step‚384€Íƒí¨‘Ü‚(:† ž†èÔ‘¾z gpu.0.gpu‚68.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚130.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HIbvÝ‚(:ë ž†èÔ ê­z! disk.nvme0n1.in‚ 21312.2734375 disk.sda2.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3263.98828125 cpu‚25.16758336396742 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 256.453125) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17 network.recv‚ 2.685205e+06) disk./.usagePercent‚77.94416728613663% disk./.usageGB‚1161.4244194030762 disk.drbd1260.out‚ 1.17578125 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚95.66822052001953% proc.memory.availableMB‚ 709.71875# disk.nvme0c0n1.in‚ 21312.2734375 disk.dm-0.in‚ 21312.2734375" disk.dm-0.out‚200254.509765625 disk.dm-1.out‚ 256.453125 disk.sda3.in‚0 disk.sda3.out‚ 256.453125 disk.drbd1260.in‚0% disk.nvme0n1.out‚200254.509765625 network.sent‚ 1.2976674e+07' disk.nvme0c0n1.out‚200254.509765625S´¯rÞ‚(:ç ­†èÔÕÉâw) disk./.usagePercent‚77.96097878889512 disk.dm-0.in‚ 21475.328125" disk.dm-0.out‚200697.189453125 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚95.66822052001953) proc.memory.percent‚19.92180347442627 cpu‚25.167801902761898 network.sent‚ 1.308749e+07 network.recv‚ 2.69172e+06" disk.nvme0c0n1.in‚ 21475.328125' disk.nvme0c0n1.out‚200697.189453125 disk.dm-1.in‚0 disk.sda.out‚ 256.890625 disk.drbd1260.out‚ 1.17578125# proc.memory.rssMB‚ 3263.98828125% disk./.usageGB‚1161.6749229431152 disk.dm-1.out‚ 256.890625 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.out‚ 256.890625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 21475.328125% disk.nvme0n1.out‚200697.189453125% proc.memory.availableMB‚ 709.71875} Èyß‚(:… ­†èÔ™Ëúw gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚226.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.3114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïéè|à‚(:ñ ¼†èÔ‡Ïåx disk.sda.in‚0 disk.sda3.out‚ 257.25390625" disk.dm-0.out‚200898.896484375 disk.drbd1260.out‚ 1.1796875$ memory_percent‚95.66972255706787 proc.cpu.threads‚17 network.recv‚ 2.695081e+06) disk./.usagePercent‚77.97768814464744 disk.nvme0n1.in‚ 21475.328125% disk.nvme0n1.out‚200898.896484375 disk.sda1.in‚0 disk.dm-1.out‚ 257.25390625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚200898.896484375 disk.dm-1.in‚0 disk.sda.out‚ 257.25390625( proc.memory.availableMB‚ 709.47265625# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 cpu‚25.167881151621586 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 21475.328125 disk.dm-0.in‚ 21475.328125 network.sent‚ 1.3092562e+07% disk./.usageGB‚1161.9239044189453 disk.sda2.out‚0¶/N˜›á‚(:‘ ¼†èÔ¤ðŠy gpu.0.gpu‚24.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112w™, gpu.0.temp‚39.0 gpu.0.powerWatts‚152.435' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.487000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à%ÿ8vâ‚(:ë ˆèÔ¥‡¦{ disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 709.4453125 network.sent‚ 1.309708e+07 network.recv‚ 2.699706e+06" disk.dm-0.out‚201149.337890625# proc.memory.rssMB‚ 3263.98828125 proc.cpu.threads‚17 disk.sda3.out‚ 257.64453125% disk./.usageGB‚1161.7949523925781 disk.dm-1.in‚0 disk.dm-1.out‚ 257.64453125 disk.sda.out‚ 257.64453125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.1796875 disk.nvme0n1.in‚ 21475.328125" disk.nvme0c0n1.in‚ 21475.328125' disk.nvme0c0n1.out‚201149.337890625% disk.nvme0n1.out‚201149.337890625 disk.sda1.out‚0$ memory_percent‚95.66988945007324) proc.memory.percent‚19.92180347442627 cpu‚25.16803841859382& disk./.usagePercent‚77.96903406596 disk.dm-0.in‚ 21475.328125eG•Ï‘ã‚(:† ˆèÔ͆ù{ gpu.0.gpu‚61.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚186.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.319 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zYÏä‚(:ô Ú†èÔüŒãx network.recv‚ 2.703146e+06) disk./.usagePercent‚77.97584924252901% disk./.usageGB‚1161.8965034484863 disk.sda2.in‚0 disk.sda1.in‚0# disk.dm-0.out‚201665.5615234375 disk.dm-1.out‚ 257.98046875 disk.sda.in‚0 network.sent‚ 1.3101928e+07 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 21475.328125( disk.nvme0c0n1.out‚201665.5615234375 disk.dm-0.in‚ 21475.328125 disk.sda3.out‚ 257.98046875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 21475.328125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 1.18359375$ memory_percent‚95.66988945007324' proc.memory.availableMB‚ 709.4453125# proc.memory.rssMB‚ 3263.98828125) proc.memory.percent‚19.92180347442627 cpu‚25.168225779696446 disk.sda1.out‚0 disk.sda.out‚ 257.98046875& disk.nvme0n1.out‚201665.5615234375 proc.cpu.threads‚17ò\N›å‚(: Ú†èÔòæ¡z gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚214.518' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.903600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>g%·æ‚(:’ é†èÔ‚–y gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚186.377' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.275400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00‡pv‡ç‚(:ü é†èÔÎäóz$ disk./.usageGB‚1162.649501800537' disk.nvme0n1.out‚201892.12841796875 disk.sda2.in‚0$ disk.dm-0.out‚201892.12841796875 disk.dm-1.out‚ 258.50390625 disk.sda.out‚ 258.50390625 disk.sda3.in‚0# memory_percent‚95.6683874130249 disk.dm-0.in‚21553.42578125 network.sent‚ 1.3108042e+07" disk.nvme0n1.in‚21553.42578125 disk.sda3.out‚ 258.50390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.18359375# proc.memory.rssMB‚ 3263.98828125 cpu‚25.168413336664173 disk.dm-1.in‚0( proc.memory.availableMB‚ 709.69140625) proc.memory.percent‚19.92180347442627 proc.cpu.threads‚17) disk.nvme0c0n1.out‚201892.12841796875 network.recv‚ 2.707442e+06) disk./.usagePercent‚78.02638359374284 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚21553.42578125 disk.sda.in‚0Ì7iÄ2è- _wandbruntime‚8529  _runtime‚8529ÖŸŽ{ÇéÁ & lr-AdamW/pg1‚0.0004959269819441372 & lr-AdamW/pg2‚0.0004959269819441372 trainer/global_step‚4749 % _timestamp‚1.7884783224429193e+09 _runtime‚8529.198824295 _step‚385þ Œ¶ÌêÆ & lr-AdamW/pg1‚0.0004959269819441372 & lr-AdamW/pg2‚0.0004959269819441372 trainer/global_step‚4749 % _timestamp‚1.7884783224429193e+09 _runtime‚8529.198824295 _step‚385¸@ðŽ2ë- _wandbruntime‚8529  _runtime‚8529ÉÔ'(ìˆP WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750834320593408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762387352030720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118683397 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11532822507048 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63158081 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15905838688256 % _timestamp‚1.7884783224443712e+09 _runtime‚8529.200120318 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99965535 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚695575783793152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750834294201856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11553057828864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118657057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15905910626304 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218648938 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚744549487958584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚762390989083136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11556668489728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63158076 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚679669821963776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚750834294201856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99965535 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚679669873166848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106288451 trainer/global_step‚4749 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11553057828864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733016665451536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118683397 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚750834320593408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99965541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43129670 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11536431169732 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024ìT¢(íP QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118683397 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚750834320593408 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99965541 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43129670 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11536431169732 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚750834320593408 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762387352030720 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118683397 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11532822507048 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63158081 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15905838688256 % _timestamp‚1.7884783224443712e+09 _runtime‚8529.200120318 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99965535 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚695575783793152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚750834294201856 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11553057828864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118657057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15905910626304 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218648938 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚744549487958584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚762390989083136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11556668489728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63158076 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚679669821963776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚750834294201856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99965535 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚679669873166848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106288451 trainer/global_step‚4749 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11553057828864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733016665451536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921‚{5šD2î- _wandbruntime‚8531  _runtime‚8531¶íøï’N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750988071760896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚762547177137152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11555468310528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚733166994692672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 trainer/global_step‚4749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218693955 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43138369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99985965 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚387 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚762543540071424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15909174957056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750988098152448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106309250 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63170881 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15909248978944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884783242580683e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750988071760896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43139047 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11535228810334 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218667583 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118707990 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99985959 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106309933 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 _runtime‚8531.015077469 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11559078984704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11559078984704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚679809112352768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚762547177137152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pÕŸ, ool.peak‚ 534404096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚733167020120640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218667583 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118681624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99985965 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚695718287309824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚744705857593866 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118681624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218693955 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750988098152448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚679809163555840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63170886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218667583 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118707990 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99985959 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106309933 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 _runtime‚8531.015077469 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11559078984704 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11559078984704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚679809112352768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚762547177137152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚733167020120640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218667583 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118681624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99985965 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚695718287309824 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚744705857593866 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118681624 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218693955 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750988098152448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚679809163555840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63170886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218693955 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43138369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99985965 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚387 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚762543540071424 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15909174957056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750988098152448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106309250 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63170881 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15909248978944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7884783242580683e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750988071760896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43139047 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11535228810334 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187ƒãE¨g2ñ- _wandbruntime‚8531  _runtime‚8531L—ÙWòÑ 1loss/FBt1/assignment_loss‚0.41566835463202423 % _timestamp‚1.7884783242608523e+09 %loss/total_loss‚48.45855562483571 epoch‚3 2loss/SRqqt3/assignment_loss‚1.9812873430826046 0loss/FRt1/assignment_loss‚1.9578755062017867 .loss/FRt1/detection_loss‚1.658835800650765 1loss/SRqqt3/detection_loss‚2.6218134749818733 .loss/FRt2/assignment_loss‚2.89247706228373 0loss/FBt3/assignment_loss‚0.5414226847351529 .loss/FRt3/detection_loss‚2.940227918624878 0loss/SRqqt4/detection_loss‚2.410756059383091 trainer/global_step‚4749 1loss/SRqqt1/assignment_loss‚1.240063196812528 .loss/FRt2/detection_loss‚2.042949942833915 /loss/FBt3/detection_loss‚2.3122320286929607 0loss/FRt3/assignment_loss‚4.9851790618896485 0loss/FRt4/assignment_loss‚3.6723837885591717 2loss/SRqqt4/assignment_loss‚1.9967672559187601 0loss/FBt4/assignment_loss‚0.3933429467433598 1loss/SRqqt1/detection_loss‚1.0577745992198306 .loss/FBt1/detection_loss‚2.928148099354335 /loss/FBt4/detection_loss‚1.8939680317416787 _runtime‚8531.016524327 0loss/FBt2/assignment_loss‚0.1602633464977201 .loss/FRt4/detection_loss‚3.149167698290613 _step‚388 2loss/SRqqt2/assignment_loss‚1.9838784947281791 1loss/SRqqt2/detection_loss‚2.3937262892723083 /loss/FBt2/detection_loss‚0.8283466397047933Ì­XÏ\óÖ 0loss/FRt1/assignment_loss‚1.9578755062017867 .loss/FRt1/detection_loss‚1.658835800650765 1loss/SRqqt3/detection_loss‚2.6218134749818733 .loss/FRt2/assignment_loss‚2.89247706228373 0loss/FBt3/assignment_loss‚0.5414226847351529 .loss/FRt3/detection_loss‚2.940227918624878 0loss/SRqqt4/detection_loss‚2.410756059383091 trainer/global_step‚4749 1loss/SRqqt1/assignment_loss‚1.240063196812528 .loss/FRt2/detection_loss‚2.042949942833915 /loss/FBt3/detection_loss‚2.3122320286929607 0loss/FRt3/assignment_loss‚4.9851790618896485 0loss/FRt4/assignment_loss‚3.6723837885591717 2loss/SRqqt4/assignment_loss‚1.9967672559187601 0loss/FBt4/assignment_loss‚0.3933429467433598 1loss/SRqqt1/detection_loss‚1.0577745992198306 .loss/FBt1/detection_loss‚2.928148099354335 /loss/FBt4/detection_loss‚1.8939680317416787 _runtime‚8531.016524327 0loss/FBt2/assignment_loss‚0.1602633464977201 .loss/FRt4/detection_loss‚3.149167698290613 _step‚388 2loss/SRqqt2/assignment_loss‚1.9838784947281791 1loss/SRqqt2/detection_loss‚2.3937262892723083 /loss/FBt2/detection_loss‚0.8283466397047933 1loss/FBt1/assignment_loss‚0.41566835463202423 % _timestamp‚1.7884783242608523e+09 %loss/total_loss‚48.45855562483571 epoch‚3 2loss/SRqqt3/assignment_loss‚1.9812873430826046„"T¸yô‚(:î ø†èԈʙx) proc.memory.percent‚19.92180347442627 cpu‚25.168542447012843 proc.cpu.threads‚17 disk.dm-1.out‚ 258.82421875 disk.sda.out‚ 258.82421875 disk.drbd1260.in‚0% disk.nvme0n1.out‚202335.810546875 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 21583.2421875' disk.nvme0c0n1.out‚202335.810546875 network.sent‚ 1.3112933e+07) disk./.usagePercent‚78.02775605013389 disk.dm-1.in‚0 disk.sda3.out‚ 258.82421875 disk.sda1.out‚0 disk.dm-0.in‚ 21583.2421875$ memory_percent‚95.67012786865234% proc.memory.availableMB‚ 709.40625 network.recv‚ 2.710911e+06% disk./.usageGB‚1162.6699523925781 disk.sda3.in‚0! disk.nvme0n1.in‚ 21583.2421875 disk.sda1.in‚0" disk.dm-0.out‚202335.810546875 disk.sda.in‚0 disk.drbd1260.out‚1.1875# proc.memory.rssMB‚ 3263.98828125ðÍÒã‘õ‚(:† ø†èԇɦy gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+RwÈxö‚(:í ‡‡èÔ’©x* proc.memory.percent‚19.922590255737305 cpu‚25.168712006639907( disk./.usagePercent‚78.0277575861791% disk./.usageGB‚1162.6699752807617 disk.dm-1.out‚ 259.3515625 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 709.52734375' disk.nvme0c0n1.out‚202633.302734375 disk.dm-0.in‚ 21583.2421875" disk.dm-0.out‚202633.302734375 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚1.1875! disk.nvme0n1.in‚ 21583.2421875 proc.cpu.threads‚17 network.recv‚ 2.716692e+06# disk.nvme0c0n1.in‚ 21583.2421875 disk.dm-1.in‚0 disk.sda.out‚ 259.3515625 disk.sda3.out‚ 259.3515625 disk.sda2.in‚0$ memory_percent‚95.66938877105713 network.sent‚ 1.3222784e+07% disk.nvme0n1.out‚202633.302734375" proc.memory.rssMB‚ 3264.1171875Œ¥i«œ÷‚(:‘ ‡‡èÔÊ×y gpu.0.gpu‚44.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚171.497' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.299400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J9­„ø‚(:ù –‡èÔù‰©x disk.dm-0.in‚ 21583.2421875 disk.dm-1.in‚0 disk.dm-1.out‚ 259.73046875 disk.sda.out‚ 259.73046875! disk.nvme0n1.in‚ 21583.2421875 disk.sda2.in‚0 disk.sda2.out‚0) disk.nvme0c0n1.out‚202941.90478515625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda1.out‚0$ memory_percent‚95.66943645477295" proc.memory.rssMB‚ 3264.1171875 cpu‚25.168910477308074' disk.nvme0n1.out‚202941.90478515625 disk.sda1.in‚0( proc.memory.availableMB‚ 709.51953125* proc.memory.percent‚19.922590255737305 proc.cpu.threads‚17 network.sent‚ 1.322592e+07 network.recv‚ 2.720056e+06% disk./.usageGB‚1162.6699752807617# disk.nvme0c0n1.in‚ 21583.2421875$ disk.dm-0.out‚202941.90478515625 disk.sda.in‚0 disk.sda3.out‚ 259.73046875 disk.drbd1260.out‚ 1.19140625( disk./.usagePercent‚78.0277575861791OÖ ’ù‚(:‡ –‡èÔ‹ ×y gpu.0.gpu‚59.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚197.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/[‘ú‚(:† ¥‡èÔÝÄ¿y gpu.0.gpu‚39.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚169.583' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œà…û‚(:ú ¥‡èÔ衦z disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 21583.2421875! disk.nvme0n1.in‚ 21583.2421875 network.recv‚ 2.724672e+06) disk./.usagePercent‚78.05987347534202 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 21583.2421875) disk.nvme0c0n1.out‚203169.43994140625 disk.dm-1.out‚ 260.24609375 disk.sda.out‚ 260.24609375 disk.sda3.in‚0$ disk.dm-0.out‚203169.43994140625 disk.sda.in‚0 disk.drbd1260.out‚ 1.19140625( proc.memory.availableMB‚ 709.27734375" proc.memory.rssMB‚ 3264.1171875 cpu‚25.169010048207966 proc.cpu.threads‚17 network.sent‚ 1.3232043e+07 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 260.24609375 disk.drbd1260.in‚0' disk.nvme0n1.out‚203169.43994140625$ memory_percent‚95.67091464996338* proc.memory.percent‚19.922590255737305$ disk./.usageGB‚1163.148525238037ÿÜc™†ü‚(:û ´‡èÔË®óx" disk.nvme0n1.in‚21728.50390625 disk.sda2.out‚0 disk.dm-1.out‚ 260.50390625$ memory_percent‚95.66941261291504 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd1260.in‚0' disk.nvme0n1.out‚203635.38623046875 disk.sda1.out‚0$ disk.nvme0c0n1.in‚21728.50390625) disk.nvme0c0n1.out‚203635.38623046875 disk.sda.out‚ 260.50390625 disk.sda3.out‚ 260.50390625$ disk.dm-0.out‚203635.38623046875 disk.dm-1.in‚0' proc.memory.availableMB‚ 709.5234375" proc.memory.rssMB‚ 3264.1171875 cpu‚25.1692157767826 network.sent‚ 1.3237043e+07 network.recv‚ 2.727685e+06% disk./.usageGB‚1163.1466941833496 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚21728.50390625* proc.memory.percent‚19.922590255737305) disk./.usagePercent‚78.05975059172557 disk.sda.in‚0 disk.drbd1260.out‚ 1.19921875˜†tY‘ý‚(:† ´‡èԼπz gpu.0.gpu‚26.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«4nˆþ‚(:ý ÇèÔìÊ’x) disk.nvme0c0n1.out‚204030.02294921875 disk.sda.out‚ 260.94921875 disk.drbd1260.out‚ 1.19921875$ memory_percent‚95.66941261291504' proc.memory.availableMB‚ 709.5234375) disk./.usagePercent‚78.05975059172557' disk.nvme0n1.out‚204030.02294921875 disk.sda2.in‚0$ disk.nvme0c0n1.in‚21728.50390625 disk.dm-0.in‚21728.50390625 disk.sda3.out‚ 260.94921875* proc.memory.percent‚19.922590255737305 cpu‚25.169359435431357% disk./.usageGB‚1163.1466941833496" disk.nvme0n1.in‚21728.50390625 disk.sda2.out‚0 disk.dm-1.out‚ 260.94921875 disk.drbd1260.in‚0" proc.memory.rssMB‚ 3264.1171875 proc.cpu.threads‚17 disk.sda1.out‚0$ disk.dm-0.out‚204030.02294921875 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 network.sent‚ 1.3243338e+07 network.recv‚ 2.732285e+06 disk.sda1.in‚0}ºäøÿ‚(:… ÇèÔµîx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WL/s‘€‚(:† Ò‡èÔ™æÙw gpu.0.gpu‚35.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.433' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4866 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰+‡ƒ‚(:ø Ò‡èÔ±ôy disk.dm-0.in‚21728.50390625 disk.dm-1.out‚ 261.4296875 disk.sda3.out‚ 261.4296875" disk.nvme0n1.in‚21728.50390625 disk.sda1.out‚0* proc.memory.percent‚19.922590255737305 network.sent‚ 1.3248226e+07 network.recv‚ 2.735703e+06) disk./.usagePercent‚78.06132555007622$ disk.nvme0c0n1.in‚21728.50390625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.203125 disk.sda1.in‚0% disk./.usageGB‚1163.1701622009277) disk.nvme0c0n1.out‚204472.95068359375 disk.sda.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 3264.1171875 cpu‚25.169516193461565 proc.cpu.threads‚17$ disk.dm-0.out‚204472.95068359375 disk.dm-1.in‚0 disk.sda.out‚ 261.4296875' disk.nvme0n1.out‚204472.95068359375 disk.sda2.in‚0$ memory_percent‚95.66941261291504' proc.memory.availableMB‚ 709.5234375ô#†2‚- _wandbruntime‚8625  _runtime‚8625Ø"þ²ÇƒÁ _runtime‚8625.465342002 _step‚389 & lr-AdamW/pg1‚0.0004918950320746624 & lr-AdamW/pg2‚0.0004918950320746624 trainer/global_step‚4799 % _timestamp‚1.7884784187092743e+09.ÛAÌ„Æ _runtime‚8625.465342002 _step‚389 & lr-AdamW/pg1‚0.0004918950320746624 & lr-AdamW/pg2‚0.0004918950320746624 trainer/global_step‚4799 % _timestamp‚1.7884784187092743e+09…âì2…- _wandbruntime‚8625  _runtime‚8625:s£ÒÁ†ŒP UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚770196768479744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚702706308931072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43570537 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚752189912861544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119915099 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚770200405532160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚758523198545408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770200405532160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚686634392616448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16072041552896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚758523172153856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 trainer/global_step‚4799 % _timestamp‚1.7884784187109911e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11677206986752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119888759 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚758523172153856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770196768479744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100986741 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16071967517696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚752186278770892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _step‚390 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220901840 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100986735 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100986741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63798331 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚702706434169344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11673596325888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119915099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43571245 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107369576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63798326 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚740533152936304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚3201÷{‡Q 02 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107368863 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11677206986752 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11673596325888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚708 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚8625.466757802 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚758523198545408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚686634341413376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11653151262556 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100986735 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11656759925240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Y&`(‡‘P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100986741 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16071967517696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚752186278770892 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _step‚390 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220901840 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100986735 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100986741 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63798331 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚702706434169344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11673596325888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119915099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43571245 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107369576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63798326 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚740533152936304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107368863 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11677206986752 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11673596325888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚708 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚8625.466757802 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚758523198545408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚686634341413376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11653151262556 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100986735 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11656759925240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚770196768479744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚702706308931072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43570537 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚752189912861544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119915099 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚770200405532160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚758523198545408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770200405532160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚686634392616448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16072041552896 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚758523172153856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 trainer/global_step‚4799 % _timestamp‚1.7884784187109911e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11677206986752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119888759 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚758523172153856 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770196768479744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346†ý F2ˆ- _wandbruntime‚8626  _runtime‚8626蛡-'‰–N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63811131 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8626.509913981 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚770356593586176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16075309281792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚770352956520448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220920485 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11659166228734 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63811136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101007159 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚758676949712896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101007165 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119939692 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11655557565842 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107390379 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101007165 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚758676976104448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚758676949712896 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚758676976104448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43579248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚702848941084160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220920485 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119939692 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚752342648406174 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚391 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220946857 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719uÒ›n'&Š›N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚758676949712896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101007165 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119939692 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11655557565842 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107390379 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101007165 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚758676976104448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚758676949712896 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚758676976104448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43579248 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚702848941084160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220920485 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119939692 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚752342648406174 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚391 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220946857 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63811131 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8626.509913981 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚770356593586176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16075309281792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚770352956520448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220920485 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11659166228734 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63811136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.sma aEØúll_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101007159 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0‡‰}™ 2‹- _wandbruntime‚8626  _runtime‚8626ê+_Q\ŒÖ 2loss/SRqqt1/assignment_loss‚1.2393787721126912 /loss/FRt1/detection_loss‚1.8574801710929023 1loss/SRqqt4/detection_loss‚2.1109706921421965 /loss/FBt3/detection_loss‚2.5356758620057787 trainer/global_step‚4799 2loss/SRqqt2/assignment_loss‚2.2239819599522486 1loss/FBt3/assignment_loss‚0.37071200162089163 _step‚392 0loss/FRt4/assignment_loss‚2.8682924463198733 1loss/FBt1/assignment_loss‚0.23968806432063636 1loss/SRqqt2/detection_loss‚3.2527250713772244 -loss/FBt4/detection_loss‚2.72140252048319 epoch‚3 1loss/SRqqt3/detection_loss‚2.9254303442107306 /loss/FRt3/assignment_loss‚4.225976042449474 1loss/FBt2/assignment_loss‚0.18986126069270423 1loss/FBt4/assignment_loss‚0.23707260570319538 /loss/FRt2/detection_loss‚1.9093422345438247 .loss/FRt3/detection_loss‚2.925128474831581 .loss/FRt4/detection_loss‚3.035269883962778 .loss/FBt2/detection_loss‚0.811913981643175 &loss/total_loss‚47.706290508761725 /loss/FRt1/assignment_loss‚2.309491255525815 0loss/FRt2/assignment_loss‚2.6113689056897567 2loss/SRqqt4/assignment_loss‚2.2940238945834013 % _timestamp‚1.7884784197565658e+09 _runtime‚8626.512567023 2loss/SRqqt3/assignment_loss‚1.5653765771400046 0loss/SRqqt1/detection_loss‚1.120825555866524 /loss/FBt1/detection_loss‚2.12490193049112943xöüaÛ /loss/FRt3/assignment_loss‚4.225976042449474 1loss/FBt2/assignment_loss‚0.18986126069270423 1loss/FBt4/assignment_loss‚0.23707260570319538 /loss/FRt2/detection_loss‚1.9093422345438247 .loss/FRt3/detection_loss‚2.925128474831581 .loss/FRt4/detection_loss‚3.035269883962778 .loss/FBt2/detection_loss‚0.811913981643175 &loss/total_loss‚47.706290508761725 /loss/FRt1/assignment_loss‚2.309491255525815 0loss/FRt2/assignment_loss‚2.6113689056897567 2loss/SRqqt4/assignment_loss‚2.2940238945834013 % _timestamp‚1.7884784197565658e+09 _runtime‚8626.512567023 2loss/SRqqt3/assignment_loss‚1.5653765771400046 0loss/SRqqt1/detection_loss‚1.120825555866524 /loss/FBt1/detection_loss‚2.1249019304911294 2loss/SRqqt1/assignment_loss‚1.2393787721126912 /loss/FRt1/detection_loss‚1.8574801710929023 1loss/SRqqt4/detection_loss‚2.1109706921421965 /loss/FBt3/detection_loss‚2.5356758620057787 trainer/global_step‚4799 2loss/SRqqt2/assignment_loss‚2.2239819599522486 1loss/FBt3/assignment_loss‚0.37071200162089163 _step‚392 0loss/FRt4/assignment_loss‚2.8682924463198733 1loss/FBt1/assignment_loss‚0.23968806432063636 1loss/SRqqt2/detection_loss‚3.2527250713772244 -loss/FBt4/detection_loss‚2.72140252048319 epoch‚3 1loss/SRqqt3/detection_loss‚2.9254303442107306ˆ¬…¬­’Ž‚(:‡ á‡èÔèö¤w gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.459' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðj……‚(:ú á‡èÔØß‹{ disk.sda2.out‚0$ disk.nvme0c0n1.in‚21728.50390625 disk.sda.in‚0 disk.sda3.out‚ 262.04296875 disk.drbd1260.in‚0' proc.memory.availableMB‚ 709.2109375" disk.nvme0n1.in‚21728.50390625 disk.sda1.in‚0 disk.dm-1.out‚ 262.04296875" proc.memory.rssMB‚ 3264.1171875 proc.cpu.threads‚17% disk./.usageGB‚1163.1684532165527' disk.nvme0n1.out‚204723.42724609375 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.out‚ 262.04296875 disk.drbd1260.out‚1.203125$ memory_percent‚95.67131996154785* proc.memory.percent‚19.922590255737305 network.recv‚ 2.740229e+06) disk./.usagePercent‚78.06121085870087) disk.nvme0c0n1.out‚204723.42724609375 disk.dm-0.in‚21728.50390625$ disk.dm-0.out‚204723.42724609375 disk.dm-1.in‚0 disk.sda3.in‚0 cpu‚25.16959001554874 network.sent‚ 1.3267403e+07³œI}‚(:ò ð‡èÔŠæÍx% disk./.usageGB‚1162.7343254089355 disk.sda.in‚0 disk.sda.out‚ 262.4296875 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚204974.0107421875 disk.sda3.in‚0 disk.sda3.out‚ 262.4296875" disk.nvme0n1.in‚21855.65234375& disk.nvme0n1.out‚204974.0107421875 disk.sda1.out‚0 disk.dm-0.in‚21855.65234375# disk.dm-0.out‚204974.0107421875 disk.dm-1.in‚0 network.recv‚ 2.745699e+06 disk.drbd1260.out‚ 1.20703125$ disk.nvme0c0n1.in‚21855.65234375 disk.dm-1.out‚ 262.4296875" proc.memory.rssMB‚ 3264.1171875* proc.memory.percent‚19.922590255737305 cpu‚ 25.1697713704 network.sent‚ 1.3361884e+07( disk./.usagePercent‚78.0320761772742$ memory_percent‚95.67115306854248( proc.memory.availableMB‚ 709.23828125 proc.cpu.threads‚17z_Æí’‘‚(:‡ ð‡èÔ€¿þx gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚167.517' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&0´’‚(:„ ÿ‡èÔÕò‹y gpu.0.gpu‚60.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚230.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0݉ÃS†“‚(:û ÿ‡èÔÏ®Íz disk.sda.in‚0% disk./.usageGB‚1162.7108573913574 disk.sda3.in‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚21855.65234375 disk.sda1.out‚0 cpu‚25.169868056783855 disk.sda3.out‚ 262.90234375& disk.nvme0n1.out‚205247.4404296875 disk.sda2.in‚0 disk.sda2.out‚0( disk.nvme0c0n1.out‚205247.4404296875# disk.dm-0.out‚205247.4404296875 disk.sda.out‚ 262.90234375$ memory_percent‚95.67115306854248 network.recv‚ 2.750504e+06$ disk.nvme0c0n1.in‚21855.65234375( proc.memory.availableMB‚ 709.23828125" proc.memory.rssMB‚ 3264.1171875* proc.memory.percent‚19.922590255737305 proc.cpu.threads‚17 network.sent‚ 1.3368285e+07) disk./.usagePercent‚78.03050121892355 disk.drbd1260.out‚ 1.20703125 disk.sda1.in‚0 disk.dm-0.in‚21855.65234375 disk.dm-1.in‚0 disk.dm-1.out‚ 262.90234375¸¡¿ì…”‚(:ú ŽˆèÔî„x" proc.memory.rssMB‚ 3264.1171875 network.sent‚ 1.3373266e+07) disk./.usagePercent‚78.03050121892355$ disk.nvme0c0n1.in‚21855.65234375 disk.sda.out‚ 263.2578125 disk.drbd1260.out‚ 1.2109375" disk.nvme0n1.in‚21855.65234375' disk.nvme0n1.out‚205666.33642578125 disk.sda2.out‚0% disk./.usageGB‚1162.7108573913574$ disk.dm-0.out‚205666.33642578125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda1.in‚0* proc.memory.percent‚19.922590255737305 cpu‚25.170076949686056 proc.cpu.threads‚17) disk.nvme0c0n1.out‚205666.33642578125 disk.sda.in‚0 disk.sda3.out‚ 263.2578125 disk.sda1.out‚0$ memory_percent‚95.67115306854248 network.recv‚ 2.753882e+06 disk.dm-0.in‚21855.65234375 disk.dm-1.out‚ 263.2578125 disk.sda3.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 709.23828125ä›gK›•‚(: ŽˆèÔñŽÚx gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚194.124' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚38.824799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÑ–‚(:… ˆèÔ¡¿Øx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.981' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5962 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k€ˆj‚—‚(:÷ ˆèÔ‡ßy$ disk.nvme0c0n1.in‚21855.65234375$ memory_percent‚95.67270278930664& proc.memory.availableMB‚ 708.984375" disk.nvme0n1.in‚21855.65234375 disk.dm-1.out‚ 263.7265625 disk.sda.out‚ 263.7265625 disk.sda3.out‚ 263.7265625) disk.nvme0c0n1.out‚206205.19970703125# proc.memory.rssMB‚ 3264.34765625* proc.memory.percent‚19.923996925354004 cpu‚25.172249729845536 network.sent‚ 1.3381332e+07 disk.sda.in‚0' disk.nvme0n1.out‚206205.19970703125 disk.sda1.out‚0 disk.dm-0.in‚21855.65234375 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.2109375 network.recv‚ 2.75839e+06) disk./.usagePercent‚78.03049993888588$ disk./.usageGB‚1162.710838317871 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚206205.19970703125 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda1.in‚0¥´®ƒ˜‚(:ø ¬ˆèÔÓµÇx proc.cpu.threads‚17 network.recv‚ 2.761891e+06 disk.sda.in‚0 disk.sda.out‚ 264.6171875 disk.drbd1260.out‚ 1.21484375' disk.nvme0n1.out‚206526.85986328125 disk.dm-0.in‚21998.33984375 disk.sda3.in‚0 disk.sda3.out‚ 264.6171875" disk.nvme0n1.in‚21998.33984375 disk.sda1.out‚0# memory_percent‚95.6740140914917) proc.memory.percent‚19.92478370666504 network.sent‚ 1.3384296e+07% disk./.usageGB‚1162.7343254089355 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚206526.85986328125( proc.memory.availableMB‚ 708.76953125 cpu‚25.178659735729067( disk./.usagePercent‚78.0320761772742$ disk.dm-0.out‚206526.85986328125 disk.dm-1.in‚0 disk.dm-1.out‚ 264.6171875 disk.sda2.in‚0$ disk.nvme0c0n1.in‚21998.33984375" proc.memory.rssMB‚ 3264.4765625ÊoÚ™‚(:… ¬ˆèÔ—¶˜y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸h;ˆš‚(:ý »ˆèÔ¿ñ¼} disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 265.04296875 proc.cpu.threads‚17 network.sent‚ 1.3392612e+07% disk./.usageGB‚1161.5378303527832 disk.sda2.out‚0 disk.drbd1260.in‚0" disk.nvme0n1.in‚21998.33984375' proc.memory.availableMB‚ 708.9765625# proc.memory.rssMB‚ 3264.57421875* proc.memory.percent‚19.925379753112793 network.recv‚ 2.766673e+06( disk./.usagePercent‚77.9517783901296 disk.sda1.in‚0 disk.dm-1.out‚ 265.04296875 disk.drbd1260.out‚ 1.21484375 cpu‚25.184558309259756 disk.sda2.in‚0$ disk.nvme0c0n1.in‚21998.33984375 disk.dm-0.in‚21998.33984375 disk.sda.in‚0 disk.sda.out‚ 265.04296875' disk.nvme0n1.out‚206740.20361328125$ memory_percent‚95.67275047302246 disk.sda1.out‚0) disk.nvme0c0n1.out‚206740.20361328125$ disk.dm-0.out‚206740.20361328125á ³Õ›‚(:… »ˆèÔ•Ê} gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚35.0 gpu.0.powerWatts‚111.657' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ze¿šœ‚(:… ʈèÔòļw gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ã2ç…‚(:ú ʈèÔÉØ¢y disk.drbd1260.out‚ 1.22265625( disk.nvme0c0n1.out‚207153.4326171875 disk.dm-1.in‚0# memory_percent‚95.6744909286499 cpu‚25.190689163470065 network.sent‚ 1.3395576e+07 disk.sda3.out‚ 265.37109375 disk.drbd1260.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚21998.33984375# disk.dm-0.out‚207153.4326171875( proc.memory.availableMB‚ 708.69140625" proc.memory.rssMB‚ 3264.6953125 network.recv‚ 2.769958e+06) disk./.usagePercent‚77.96252814649235& disk.nvme0n1.out‚207153.4326171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚21998.33984375 disk.dm-1.out‚ 265.37109375 disk.sda.in‚0" disk.nvme0n1.in‚21998.33984375 disk.sda2.in‚0 disk.sda.out‚ 265.37109375* proc.memory.percent‚19.926118850708008 proc.cpu.threads‚17% disk./.usageGB‚1161.6980094909668 disk.sda3.in‚0V‘)€ž‚(:õ ÙˆèÔ”üŽx( disk./.usagePercent‚77.9609539561643" disk.nvme0n1.in‚21998.33984375& disk.nvme0n1.out‚207499.1396484375 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚95.67451477050781" proc.memory.rssMB‚ 3264.8359375 proc.cpu.threads‚17 network.recv‚ 2.774408e+06% disk./.usageGB‚1161.6745529174805 disk.dm-1.out‚ 265.78515625 disk.sda.in‚0 disk.drbd1260.out‚ 1.22265625 disk.sda1.out‚0$ proc.memory.availableMB‚708.6875* proc.memory.percent‚19.926977157592773 network.sent‚ 1.3401708e+07( disk.nvme0c0n1.out‚207499.1396484375 disk.dm-1.in‚0 disk.sda.out‚ 265.78515625 disk.sda3.in‚0 cpu‚25.19759969949673$ disk.nvme0c0n1.in‚21998.33984375 disk.dm-0.in‚21998.33984375# disk.dm-0.out‚207499.1396484375 disk.sda3.out‚ 265.78515625 disk.drbd1260.in‚0 disk.sda2.in‚0•!jŸ‚(:„ ÙˆèÔçí¯x gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.649 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0› Ž› ‚(:ô èˆèÔÛÌ´x disk.sda2.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 22129.5078125( proc.memory.availableMB‚ 708.44140625* proc.memory.percent‚19.927096366882324 network.recv‚ 2.777379e+06% disk.nvme0n1.out‚207868.982421875' disk.nvme0c0n1.out‚207868.982421875 disk.dm-0.in‚ 22129.5078125$ memory_percent‚95.67601680755615 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 266.24609375! disk.nvme0n1.in‚ 22129.5078125 disk.sda1.out‚0" disk.dm-0.out‚207868.982421875# proc.memory.rssMB‚ 3264.85546875 proc.cpu.threads‚17 network.sent‚ 1.3406593e+07) disk./.usagePercent‚77.96410668894846 disk.dm-1.out‚ 266.24609375 disk.sda.out‚ 266.24609375 disk.sda3.in‚0 disk.drbd1260.out‚ 1.2265625 cpu‚25.20309738043651% disk./.usageGB‚1161.7215309143066 disk.drbd1260.in‚0 disk.sda2.in‚0ÄŸùÍ‘¡‚(:† èˆèÔµ³»x gpu.0.gpu‚59.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚211.819' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.3638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éxÛÁ2¢- _wandbruntime‚8784  _runtime‚8784¢„¼{z*£ôT TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2732853760 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚769487157204992 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚769487157204992 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 223757793 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20708 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 130984448 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 33395048448 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2187 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2706586280 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32002 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3896742400 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3921 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 121252924 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚762766115041486 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11772714454710 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20708 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29392819712 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 121232216 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880505161 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2732853760 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2708559360 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 223737080 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚877 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚16222460093440 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3719 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚750996107173056 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2187 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11790686970368 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚16222329108992 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 102504864 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11793395529728 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚762768844958582 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚64876874 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1532 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚781277819880960 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 102504869 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 171701760 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 4586471424 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 109029614 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223757793 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 223737080 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 109029802 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2090 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3719 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚781280552734720 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚781280552734720 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚696452464200704 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2187 _step‚393 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32002 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚188 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2729917096 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4507766904 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 33395048448 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708559360 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚11770007868430 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11793395529728 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚185 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚44152743 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚696452423483392 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3719 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 121252924 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 121232216 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚769487132910592 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚712674752592384 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1532 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚712674924294144 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚64876871 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚8784.186078438ûjFâ¤ùT TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 33395048448 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2187 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2706586280 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32002 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3896742400 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3921 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 121252924 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚5 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚762766115041486 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚11772714454710 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20708 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 29392819712 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 121232216 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880505161 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2732853760 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2708559360 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 223737080 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚877 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚16222460093440 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3719 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚750996107173056 PGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚2187 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11790686970368 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚16222329108992 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 102504864 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚11793395529728 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚762768844958582 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚64876874 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1532 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚781277819880960 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 102504869 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 171701760 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 4586471424 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 109029614 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223757793 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 223737080 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 109029802 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2090 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3719 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚781280552734720 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚781280552734720 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚696452464200704 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2187 _step‚393 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32002 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚188 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2729917096 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 4507766904 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 33395048448 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708559360 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚11770007868430 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11793395529728 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚185 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚44152743 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚696452423483392 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚3719 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 121252924 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 121232216 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚769487132910592 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚712674752592384 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1532 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚712674924294144 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚64876871 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚8784.186078438 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2732853760 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚769487157204992 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚769487157204992 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 223757793 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20708 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 130984448 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0‰²ÑÉ2¥- _wandbruntime‚8785  _runtime‚8785m™„)¦þR NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20708 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚2090 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 NEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚2187 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3921 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 130984448 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚3719 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 25702182912 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 121240946 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚696547613834240 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223761370 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 33395048448 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11791237548544 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚877 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚712770493521408 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223782083 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223761370 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚769594769420288 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 102520424 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3719 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚64889213 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚44158115 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚44157930 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3719 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507766904 trainer/global_step‚4835 _step‚394 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚11793946107904 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2708559360 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 109047143 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚40717312 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 28808577024 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚762869410134170 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11793946107904 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 121261654 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚762866680217074 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2732853760 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1532 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 102520429 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 33395048448 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚769594793714688 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32002 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 29392819712 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11791237548544 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3921 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚185 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 223782083 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223761370 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚769594769420288 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 102520424 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3719 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚64889213 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚44158115 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚44157930 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3719 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507766904 trainer/global_step‚4835 _step‚394 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚11793946107904 \MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 2708559360 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 109047143 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚40717312 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 28808577024 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚762869410134170 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11793946107904 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 121261654 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚762866680217074 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 2732853760 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1532 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 102520429 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 33395048448 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚769594793714688 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32002 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 29392819712 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11791237548544 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3921 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚185 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23168006539344788 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.27718859910964966 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2973684072494507 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18706895411014557 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_2_of_7‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5031388998031616 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2371259182691574 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990662932395935 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.27348482608795166 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.010849249549210072 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.27631983160972595 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39798852801322937 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011776753701269627 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5842474102973938 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3340720236301422 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995331764221191 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6430531740188599 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979071021080017 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2659724950790405 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5466626882553101 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.32237884402275085 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005777636077255011 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2760034203529358 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08812446147203445 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31834760308265686 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.253313809633255 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884785947622433e+09 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_6‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13134920597076416 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 jet/accuracy_5_of_6‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3224988281726837 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854070544242859 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 particle/accuracy_4_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35090482234954834 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5043859481811523 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.24975436925888062 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.31535375118255615 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.37387487292289734 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_7‚0.5 particle/accuracy_1_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 +jet/accuracy_2_of_6‚0.46794870495796204 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.336643785238266 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9934558272361755 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8060888648033142 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26982295513153076 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24404653906822205 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6267857551574707 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6955739855766296 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9944444894790649 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.39262688159942627 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3224988281726837 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 +jet/accuracy_2_of_3‚0.23573251068592072 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02173091471195221 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41784408688545227 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333134651184 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899925589561462 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6001983880996704 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8402002453804016 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882278442382812 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8932708501815796 particle/accuracy_1_of_2‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.28115466237068176 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.023693526163697243 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2262062132358551 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9883041381835938 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10631079971790314 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2817701995372772 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012407851405441761 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3436756134033203 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_3_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5632173418998718 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4444444477558136 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27373987436294556 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16808436810970306 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 _runtime‚8801.520207785 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906917214393616 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3405373990535736 *jet/accuracy_1_of_2‚0.6923883557319641 'particle/f_score‚0.3788124918937683 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9809626340866089 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4072456955909729 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.34991753101348877 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.4166666567325592 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39444446563720703 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008696192875504494 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1351417452096939 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.362753301858902 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3828545808792114 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.003206669818609953 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.31333333253860474 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29281914234161377 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 )jet/accuracy_1_of_3‚0.817979097366333 +particle/specificity‚0.9378980994224548 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9816436171531677 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2824834883213043 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5889941453933716 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_7_of_7‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.28132423758506775 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.29312169551849365 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2546296715736389 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23555557429790497 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901795387268066 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.21502059698104858 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.6000000238418579 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19154760241508484 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.531038761138916 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27398473024368286 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1024603396654129 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3919755220413208 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4015173614025116 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9910609722137451 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_5‚0.9050115346908569 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2613201439380646 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2641731798648834 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3215237259864807 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879314303398132 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991787850856781 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2574324309825897 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 *loss/val_total_loss‚46.856524579465095 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013271604664623737 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092589259147644 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02572539448738098 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2585993707180023 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10316642373800278 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2476712018251419 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30133581161499023 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2742972671985626 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 epoch‚3 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2142108529806137 particle/accuracy_5_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3366071283817291 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_3_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921286702156067 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9897895455360413 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 particle/accuracy_6_of_6‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2916666269302368 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0535714291036129 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930970668792725 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4661298394203186 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4417675733566284 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5278161764144897 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2113095223903656 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚4835 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.509121835231781 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3773190677165985 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_1_of_6‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333134651184 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2438536286354065 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.25426098704338074 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.26606130599975586 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.43842360377311707 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.33796295523643494 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4633333086967468 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32932448387145996 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989405870437622 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3272119462490082 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5203440189361572 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210526347160339 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.36227405071258545 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.029109584167599678 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 particle/accuracy_4_of_5‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 particle/accuracy_2_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3227388858795166 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 _step‚395 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.24925675988197327 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.520948588848114 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941977262496948 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19968000054359436 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05425417050719261 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21651548147201538 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2332242876291275 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5488631129264832 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015249432995915413 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5508241653442383 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.11538461595773697 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015616918914020061 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 -jet/accuracy_4_of_4‚0.0021919880528002977 particle/accuracy_1_of_3‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16197451949119568 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3994268476963043 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2336336076259613 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.31586310267448425 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9937106966972351 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_3‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4126041829586029 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.010192740708589554 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3901858329772949 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2839869558811188 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.4166666567325592 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.39102253317832947 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.043575920164585114 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40321752429008484 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08553600311279297 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2782970070838928 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.20976433157920837 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969722032546997 *jet/accuracy_1_of_1‚0.4127139747142792 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3525679111480713 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4208163022994995 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.27623453736305237 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3711440861225128 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1710938662290573 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2147093117237091 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9898701310157776 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4593023359775543 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4394022524356842 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2270115315914154œ‰Üù¬¢¥ 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.23555557429790497 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901795387268066 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.21502059698104858 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.6000000238418579 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19154760241508484 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.531038761138916 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27398473024368286 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1024603396654129 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.08771929889917374 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3919755220413208 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4015173614025116 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9910609722137451 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_5‚0.9050115346908569 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.2613201439380646 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2641731798648834 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3215237259864807 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879314303398132 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991787850856781 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2574324309825897 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1785714328289032 *loss/val_total_loss‚46.856524579465095 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013271604664623737 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092589259147644 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.02572539448738098 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2585993707180023 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10316642373800278 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2476712018251419 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30133581161499023 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2742972671985626 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 epoch‚3 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 <"Purity/2FRt1SRqqt1FBt/event_purity‚0.0059523810632526875 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2142108529806137 particle/accuracy_5_of_5‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3366071283817291 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_3_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921286702156067 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9897895455360413 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 particle/accuracy_6_of_6‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.2916666269302368 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0535714291036129 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930970668792725 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4661298394203186 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4417675733566284 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5278161764144897 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2113095223903656 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚4835 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.509121835231781 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3773190677165985 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_1_of_6‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333134651184 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2438536286354065 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.25426098704338074 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.26606130599975586 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.43842360377311707 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.33796295523643494 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4633333086967468 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32932448387145996 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989405870437622 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3272119462490082 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5203440189361572 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210526347160339 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.36227405071258545 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.029109584167599678 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 particle/accuracy_4_of_5‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 particle/accuracy_2_of_7‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3227388858795166 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 _step‚395 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.24925675988197327 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.520948588848114 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941977262496948 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19968000054359436 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05425417050719261 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21651548147201538 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2332242876291275 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5488631129264832 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015249432995915413 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5508241653442383 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.11538461595773697 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015616918914020061 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 -jet/accuracy_4_of_4‚0.0021919880528002977 particle/accuracy_1_of_3‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16197451949119568 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3994268476963043 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2336336076259613 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.31586310267448425 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9937106966972351 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_3‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +jet/accuracy_4_of_6‚0.01923076994717121 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4126041829586029 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.010192740708589554 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3901858329772949 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.2839869558811188 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.4166666567325592 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.39102253317832947 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.043575920164585114 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40321752429008484 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_4‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08553600311279297 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2782970070838928 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.20976433157920837 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9969722032546997 *jet/accuracy_1_of_1‚0.4127139747142792 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3525679111480713 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4208163022994995 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.27623453736305237 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3711440861225128 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1710938662290573 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2147093117237091 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9898701310157776 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4593023359775543 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4394022524356842 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2270115315914154 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4131945073604584 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5074892044067383 +jet/accuracy_2_of_2‚0.10128346085548401 7 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.6000000238418579 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22905650734901428 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15644459426403046 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.40366342663764954 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5371527075767517 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +particle/sensitivity‚0.3122512102127075 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7368420958518982 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056738138198853 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9956473708152771 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3561052680015564 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365078806877136 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021551724523305893 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4364631175994873 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278960824012756 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_6‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.14930956065654755 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_3‚0.02262793481349945 particle/accuracy_3_of_6‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2372700423002243 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.49281224608421326 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19154760241508484 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9670138955116272 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.28611111640930176 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019726131111383438 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17394322156906128 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3517528176307678 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4725574851036072 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3413954973220825 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5448192954063416 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2949714660644531 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.27638497948646545 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.25297945737838745 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.32962584495544434 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990662932395935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24355889856815338 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23168006539344788 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.27718859910964966 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.2973684072494507 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18706895411014557 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_2_of_7‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5031388998031616 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2371259182691574 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990662932395935 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.27348482608795166 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.010849249549210072 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.27631983160972595 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.39798852801322937 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.15625 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011776753701269627 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5842474102973938 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3340720236301422 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995331764221191 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6430531740188599 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979071021080017 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2659724950790405 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5466626882553101 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.32237884402275085 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005777636077255011 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2760034203529358 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08812446147203445 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31834760308265686 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.253313809633255 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 % _timestamp‚1.7884785947622433e+09 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_6‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13134920597076416 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 jet/accuracy_5_of_6‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3224988281726837 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854070544242859 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 particle/accuracy_4_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35090482234954834 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5043859481811523 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.24975436925888062 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.31535375118255615 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.37387487292289734 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_7‚0.5 particle/accuracy_1_of_7‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 +jet/accuracy_2_of_6‚0.46794870495796204 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.336643785238266 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9934558272361755 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8060888648033142 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26982295513153076 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24404653906822205 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6267857551574707 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2291666716337204 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6955739855766296 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9944444894790649 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.39262688159942627 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3224988281726837 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 +jet/accuracy_2_of_3‚0.23573251068592072 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02173091471195221 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.41784408688545227 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333134651184 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9899925589561462 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6001983880996704 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8402002453804016 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882278442382812 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8932708501815796 particle/accuracy_1_of_2‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.28115466237068176 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.023693526163697243 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2262062132358551 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9883041381835938 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10631079971790314 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2817701995372772 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Pur¦¯óõ°ity/2FRt4SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012407851405441761 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3436756134033203 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_3_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5632173418998718 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4444444477558136 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27373987436294556 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16808436810970306 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 _runtime‚8801.520207785 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906917214393616 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3405373990535736 *jet/accuracy_1_of_2‚0.6923883557319641 'particle/f_score‚0.3788124918937683 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9809626340866089 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4072456955909729 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.34991753101348877 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.4166666567325592 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39444446563720703 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008696192875504494 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1351417452096939 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.362753301858902 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3828545808792114 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.003206669818609953 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.31333333253860474 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29281914234161377 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 )jet/accuracy_1_of_3‚0.817979097366333 +particle/specificity‚0.9378980994224548 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9816436171531677 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2824834883213043 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5889941453933716 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_7_of_7‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.28132423758506775 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.29312169551849365 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2546296715736389 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086‹.u¸Îw­‚(:ì †‰èÔíÒ¨x disk.sda1.out‚0# disk.nvme0c0n1.in‚ 22129.5078125& disk.nvme0c0n1.out‚208274.41015625 disk.dm-0.in‚ 22129.5078125! disk.dm-0.out‚208274.41015625$ disk.nvme0n1.out‚208274.41015625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 267.1796875 disk.sda3.out‚ 267.1796875$ memory_percent‚99.97613430023193& proc.memory.availableMB‚ 3.91015625# proc.memory.rssMB‚ 5297.49609375 network.recv‚ 2.787427e+06 disk.drbd1260.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚77.92198423329471! disk.nvme0n1.in‚ 22129.5078125 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 267.1796875 disk.sda.in‚0) proc.memory.percent‚32.33335018157959 cpu‚25.21704652201166 network.sent‚ 1.3520434e+07% disk./.usageGB‚1161.0938758850098 disk.drbd1260.out‚ 1.23046875®¬C®‚(:„ †‰èÔŠÕÍx gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚33.0 gpu.0.powerWatts‚94.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.9028 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óó^››¯‚(: •‰èÔÆÕ†x gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚232.985' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚46.59700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý# ˜ƒ°‚(:ø •‰èÔ¾¯îy network.sent‚ 1.3644492e+07 disk.sda.in‚0 disk.sda3.out‚ 267.6484375 disk.dm-1.in‚0* proc.memory.percent‚19.986987113952637 disk.sda.out‚ 267.6484375 disk.sda2.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 22129.5078125) disk.nvme0c0n1.out‚216354.32568359375 disk.dm-0.in‚ 22129.5078125 disk.dm-1.out‚ 267.6484375# proc.memory.rssMB‚ 3274.66796875 network.recv‚ 2.793811e+06( disk./.usagePercent‚77.9996172380154% disk./.usageGB‚1162.2506637573242 disk.sda3.in‚0 disk.drbd1260.out‚ 1.23046875 disk.sda1.in‚0$ memory_percent‚68.25988292694092 cpu‚25.21541055582528 disk.drbd1260.in‚0$ disk.dm-0.out‚216342.70068359375) proc.memory.availableMB‚ 5200.30078125 proc.cpu.threads‚17! disk.nvme0n1.in‚ 22129.5078125' disk.nvme0n1.out‚216354.45068359375 disk.sda2.in‚0ƒñ£02±- _wandbruntime‚8830  _runtime‚8830ÿ ƲÀ & lr-AdamW/pg2‚0.0004878284945180082 trainer/global_step‚4849 $ _timestamp‚1.788478623489944e+09 _runtime‚8830.245615272 _step‚396 & lr-AdamW/pg1‚0.0004878284945180082 nV˳Š& lr-AdamW/pg2‚0.0004878284945180082 trainer/global_step‚4849 $ _timestamp‚1.788478623489944e+09 _runtime‚8830.245615272 _step‚396 & lr-AdamW/pg1‚0.0004878284945180082Œhï2´- _wandbruntime‚8830  _runtime‚8830ñãJ(µ‹P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65216185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65216180 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 _runtime‚8830.246667445 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699595883202560 trainer/global_step‚4849 _step‚397 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699595934405632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11811688877338 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121672388 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102988215 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚772993103597568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 112655360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121698728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121672388 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102988221 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚754350612537968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102988221 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚754350637965936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚784821941836288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61452288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102988215 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121698728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11828838238720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚784825578888704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚784821941836288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224686949 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772993129989120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715870519819776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11828838238720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11808080214654 $ _timestamp‚1.788478623490805e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11832448899584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚766158692752622 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719ѵTÚ(¶P KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65216185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65216180 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 _runtime‚8830.246667445 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699595883202560 trainer/global_step‚4849 _step‚397 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699595934405632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11811688877338 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121672388 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102988215 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚772993103597568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 112655360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121698728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121672388 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102988221 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚754350612537968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102988221 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚754350637965936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚784821941836288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚61452288 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102988215 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121698728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11828838238720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚784825578888704 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚784821941836288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224686949 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772993129989120 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715870519819776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11828838238720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11808080214654 $ _timestamp‚1.788478623490805e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11832448899584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚766158692752622 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024»ñ·2·- _wandbruntime‚8831  _runtime‚8831Š¹â€ƒ¸–N FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121696955 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11831248720384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _runtime‚8831.341417526 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11834859394560 % _timestamp‚1.7884786245850163e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109577898 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11814095180832 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚773146881156608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61438976 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109577258 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224731966 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103008639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65228985 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716013240869888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699735224794624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚784978129876992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103008645 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224731966 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65228990 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103008639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65228985 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚716013240869888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699735224794624 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚784978129876992 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103008645 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224731966 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65228990 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121696955 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11831248720384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 _runtime‚8831.341417526 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11834859394560 % _timestamp‚1.7884786245850163e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109577898 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11814095180832 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚773146881156608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚61438976 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109577258 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224731966 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104009415 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚792631358285312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚706560402652160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110639797 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122904090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚780682007941120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚792631358285312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226939851 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65856435 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11949376735744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226913505 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚723002944600576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16442418807296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚792634995337728#„ÊŠ(ÌP WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚780682007941120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚780681981549568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11949376735744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104009421 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104009415 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚792631358285312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚706560402652160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110639797 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122904090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚780682007941120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚792631358285312 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226939851 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65856435 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11949376735744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226913505 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚723002944600576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16442418807296 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚792634995337728 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚723002821459456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5‘d¹Ó{2Í- _wandbruntime‚8909  _runtime‚8909q¯ÀK'ΖN I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚780835785500160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762017403835904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884787030338333e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44792144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69827584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762017429263872 trainer/global_step‚4899 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11934423936340 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚780835759108608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚792791183391744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚723145464380416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122928657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65869235 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226984868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚792787546326016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104029839 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121030656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11951787217408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226958496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104029845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16445771339264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚780835785500160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚792791183391744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226984868 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110661379 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚773948219109352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11930815273448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424Â:ûôÈ#Ï›N JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762017429263872 trainer/global_step‚4899 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11934423936340 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚780835759108608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚792791183391744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚723145464380416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122928657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65869235 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226984868 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚792787546326016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104029839 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121030656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11951787217408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226958496 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104029845 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16445771339264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚780835785500160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚792791183391744 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226984868 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110661379 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚773948219109352 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11930815273448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚780835785500160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚aÊÖ>Y 28880505161 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762017403835904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884787030338333e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44792144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69827584’›‘ 2Ð- _wandbruntime‚8909  _runtime‚8909h? VÑÐ $ _timestamp‚1.788478703036116e+09 _runtime‚8909.791648801 /loss/FBt4/detection_loss‚2.1139432254590487 0loss/FBt4/assignment_loss‚0.6034241799804333 1loss/FBt1/assignment_loss‚0.15902863460360095 1loss/SRqqt3/detection_loss‚2.4796297512948513 /loss/FBt1/detection_loss‚2.8794617392122746 /loss/FRt3/assignment_loss‚3.775655245780945 .loss/FRt3/detection_loss‚3.484013601144155 /loss/FRt2/detection_loss‚1.9406496653954188 .loss/FRt2/assignment_loss‚2.61796535551548 2loss/SRqqt1/assignment_loss‚1.2811414927671678 1loss/FBt2/assignment_loss‚0.11837062220785205 0loss/SRqqt4/detection_loss‚2.115089578506274 .loss/FBt2/detection_loss‚1.031318999024538 .loss/FBt3/detection_loss‚2.586445164680481 1loss/SRqqt4/assignment_loss‚2.038552020700314 %loss/total_loss‚46.49938698474786 .loss/FRt1/detection_loss‚1.811128702442122 /loss/FRt1/assignment_loss‚2.480883899381605 1loss/SRqqt2/assignment_loss‚1.751642372678308 2loss/SRqqt3/assignment_loss‚1.6383477663621306 _step‚403 1loss/FBt3/assignment_loss‚0.11558713968649197 /loss/FRt4/detection_loss‚2.6858826180299125 1loss/SRqqt2/detection_loss‚2.7951025822583366 epoch‚4 trainer/global_step‚4899 0loss/FRt4/assignment_loss‚3.0498493115107217 1loss/SRqqt1/detection_loss‚0.9462733161254008œ~oy[ÒÕ 1loss/SRqqt2/assignment_loss‚1.751642372678308 2loss/SRqqt3/assignment_loss‚1.6383477663621306 _step‚403 1loss/FBt3/assignment_loss‚0.11558713968649197 /loss/FRt4/detection_loss‚2.6858826180299125 1loss/SRqqt2/detection_loss‚2.7951025822583366 epoch‚4 trainer/global_step‚4899 0loss/FRt4/assignment_loss‚3.0498493115107217 1loss/SRqqt1/detection_loss‚0.9462733161254008 $ _timestamp‚1.788478703036116e+09 _runtime‚8909.791648801 /loss/FBt4/detection_loss‚2.1139432254590487 0loss/FBt4/assignment_loss‚0.6034241799804333 1loss/FBt1/assignment_loss‚0.15902863460360095 1loss/SRqqt3/detection_loss‚2.4796297512948513 /loss/FBt1/detection_loss‚2.8794617392122746 /loss/FRt3/assignment_loss‚3.775655245780945 .loss/FRt3/detection_loss‚3.484013601144155 /loss/FRt2/detection_loss‚1.9406496653954188 .loss/FRt2/assignment_loss‚2.61796535551548 2loss/SRqqt1/assignment_loss‚1.2811414927671678 1loss/FBt2/assignment_loss‚0.11837062220785205 0loss/SRqqt4/detection_loss‚2.115089578506274 .loss/FBt2/detection_loss‚1.031318999024538 .loss/FBt3/detection_loss‚2.586445164680481 1loss/SRqqt4/assignment_loss‚2.038552020700314 %loss/total_loss‚46.49938698474786 .loss/FRt1/detection_loss‚1.811128702442122 /loss/FRt1/assignment_loss‚2.480883899381605“q#)¨œÓ‚(:‘ ï‰èÔ‰õñv gpu.0.gpu‚44.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚268.047' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.60940000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÛqŸÔ‚(:” ï‰èÔ樊{ disk.sda.out‚ 270.42578125 disk.drbd1260.out‚ 1.2421875" disk.nvme0n1.in‚22414.50390625 network.sent‚ 1.3779815e+07 disk.sda2.in‚0 disk.dm-0.in‚22414.50390625 disk.drbd1260.in‚0' disk.nvme0n1.out‚218113.90380859375( proc.memory.availableMB‚ 5199.8359375# proc.memory.rssMB‚ 3274.73046875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 270.42578125$ memory_percent‚68.26272010803223 cpu‚25.2152770988271 proc.cpu.threads‚17 network.recv‚ 2.819324e+06 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 270.42578125( proc.memory.percent‚19.9873685836792) disk./.usagePercent‚77.99994569568184$ disk./.usageGB‚1162.255558013916$ disk.nvme0c0n1.in‚22414.50390625) disk.nvme0c0n1.out‚218113.90380859375$ disk.dm-0.out‚218113.90380859375iL”ÊÕ‚(:’ þ‰èÔ¸®w gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚241.514' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.302800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OtÓ’Ö‚(:‡ þ‰èÔ¹î»x disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 5199.94140625# proc.memory.rssMB‚ 3274.73046875 network.recv‚ 2.824632e+06 disk.dm-0.out‚ 218341.390625 disk.drbd1260.out‚ 1.24609375 disk.sda2.in‚0 network.sent‚ 1.3888171e+07 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0$ memory_percent‚68.26207637786865( proc.memory.percent‚19.9873685836792 cpu‚25.21523922695073 proc.cpu.threads‚17' disk./.usagePercent‚78.008571357533% disk./.usageGB‚1162.3840866088867$ disk.nvme0c0n1.out‚ 218341.390625 disk.dm-1.out‚ 270.79296875 disk.sda.in‚ 0.00390625 disk.sda.out‚ 270.79296875 disk.sda3.out‚ 270.79296875 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚22414.50390625 disk.dm-0.in‚22414.50390625" disk.nvme0n1.in‚22414.50390625" disk.nvme0n1.out‚ 218341.390625”›ó ‘ׂ(:† ŠèÔå½Óy gpu.0.gpu‚68.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.927' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.9854 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢PFk’Ø‚(:‡ ŠèÔñòÏ{ disk.sda.out‚ 271.203125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.24609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0% disk./.usageGB‚1162.3840713500977% disk.nvme0c0n1.out‚218580.9765625 disk.dm-0.out‚218580.9765625$ memory_percent‚82.16450214385986# proc.memory.rssMB‚ 3274.73046875 cpu‚25.215223281015067 network.recv‚ 2.829305e+06 disk.dm-1.out‚ 271.203125 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 271.203125 disk.sda1.in‚0$ disk.nvme0c0n1.in‚22414.50390625 proc.cpu.threads‚17) disk./.usagePercent‚78.00857033350286" disk.nvme0n1.in‚22414.50390625# disk.nvme0n1.out‚218580.9765625 disk.dm-0.in‚22414.50390625) proc.memory.availableMB‚ 2922.16796875( proc.memory.percent‚19.9873685836792 network.sent‚ 1.3895795e+07 disk.dm-1.in‚ 0.00390625Ó‘BkÙ‚(:’ œŠèÔª³x gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚243.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.782199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šL¿LŒÚ‚(: œŠèÔÿ¹ðx disk.drbd1260.out‚1.25" disk.nvme0n1.in‚22765.72265625 disk.sda1.in‚0' proc.memory.availableMB‚ 10.77734375% disk.nvme0n1.out‚218830.693359375 disk.sda2.out‚0# proc.memory.rssMB‚ 3274.73046875 proc.cpu.threads‚17 disk.sda3.in‚ 0.00390625 network.sent‚ 1.3898717e+07 network.recv‚ 2.832664e+06( disk./.usagePercent‚78.0127598968007% disk./.usageGB‚1162.4464988708496 disk.dm-0.in‚22765.72265625" disk.dm-0.out‚218830.693359375 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚22765.72265625 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 271.84375 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚99.93422031402588( proc.memory.percent‚19.9873685836792 cpu‚25.215241527611305 disk.sda.in‚ 0.00390625 disk.sda.out‚ 271.84375' disk.nvme0c0n1.out‚218830.693359375 disk.dm-1.out‚ 271.84375覚“Û‚(:ˆ «ŠèÔÈ£×w$ disk.nvme0c0n1.in‚22900.91015625 disk.dm-1.out‚ 272.3359375 disk.sda.out‚ 272.3359375( proc.memory.percent‚19.9873685836792 cpu‚25.215250905455143 proc.cpu.threads‚17" disk.nvme0n1.in‚22900.91015625 disk.sda2.out‚0' disk.nvme0c0n1.out‚218983.990234375 disk.sda.in‚ 0.00390625) disk./.usagePercent‚78.01695458024923 disk.drbd1260.out‚1.25 disk.sda1.out‚0 disk.dm-0.in‚22900.91015625 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 272.3359375$ memory_percent‚99.93422031402588 network.sent‚ 1.3905177e+07 network.recv‚ 2.837614e+06% disk./.usageGB‚1162.5090026855469 disk.drbd1260.in‚0 disk.sda1.in‚0" disk.dm-0.out‚218983.990234375' proc.memory.availableMB‚ 10.77734375# proc.memory.rssMB‚ 3274.73046875% disk.nvme0n1.out‚218983.990234375 disk.sda2.in‚0¾%®N’Ü‚(:‡ «ŠèÔø¬Îw gpu.0.gpu‚75.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚141.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QW‘Ý‚(:† ºŠèÔÃÜÒx gpu.0.gpu‚60.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚116.492' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;U˜Þ‚(: ºŠèÔÝãˆz" disk.nvme0n1.in‚22900.91015625 disk.sda.out‚ 272.8203125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.25390625 disk.sda2.in‚0 disk.sda1.in‚0# memory_percent‚99.9342679977417' proc.memory.availableMB‚ 10.76953125 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 272.8203125 network.recv‚ 2.840647e+06 disk.dm-1.out‚ 272.8203125% disk.nvme0n1.out‚219146.853515625# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 cpu‚25.215267108114688' disk.nvme0c0n1.out‚219146.853515625" disk.dm-0.out‚219146.853515625 disk.sda2.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.00390625 network.sent‚ 1.3910146e+07) disk./.usagePercent‚78.02429994842164% disk./.usageGB‚1162.6184539794922$ disk.nvme0c0n1.in‚22900.91015625 disk.dm-0.in‚22900.91015625 disk.dm-1.in‚ 0.00390625}Ç—2ß- _wandbruntime‚8995  _runtime‚8995‘‰¦ÇàÁ trainer/global_step‚4949 % _timestamp‚1.7884787883917592e+09 _runtime‚8995.147627071 _step‚404 & lr-AdamW/pg1‚0.0004795950532159014 & lr-AdamW/pg2‚0.0004795950532159014¨éÌáÆ & lr-AdamW/pg1‚0.0004795950532159014 & lr-AdamW/pg2‚0.0004795950532159014 trainer/global_step‚4949 % _timestamp‚1.7884787883917592e+09 _runtime‚8995.147627071 _step‚404”[ë:›2â- _wandbruntime‚8995  _runtime‚8995Ê/È›(ãŠP WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12069915232768 % _timestamp‚1.7884787883932679e+09 _step‚405 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105030621 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124162132 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚769383562079536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚800444411786752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚781435908467890 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚788370885893120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚781432274377238 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12069915232768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105030615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚800440774734336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚788370859501568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124135792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 8995.1492079 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105030615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45227911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚730134327471104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111723885 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66496680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105030621 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚713524922101760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12052346388354 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12073525893632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12048737725670 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111724596 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16609405369344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚788370885893120 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚713524973304832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚788370859501568 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚800440774734336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229192753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12073525893632s›mÑ(äP M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124135792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚730134450612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105030615 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45227911 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚730134327471104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111723885 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66496680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105030621 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚713524922101760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12052346388354 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12073525893632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12048737725670 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111724596 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16609405369344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚788370885893120 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚713524973304832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚788370859501568 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚800440774734336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229192753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12073525893632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12069915232768 % _timestamp‚1.7884787883932679e+09 _step‚405 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105030621 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124162132 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚769383562079536 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚800444411786752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚781435908467890 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚788370885893120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚711 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚781432274377238 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12069915232768 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105030615 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚800440774734336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚788370859501568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124135792 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚ 8995.1492079 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105051039 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124186725 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229211398 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚713664212490752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111745472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚713664263693824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚788524637060608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚730276939283456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229237770 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884787894041574e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45235987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12054752691848 trainer/global_step‚4949 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105051045 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12051144028956 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788524663452160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12072325714432 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124160359 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚781588644012520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105051045 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124160359 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124186725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45236665 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _step‚406 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12075936388608 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12072325714432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105051039 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105051039 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124186725 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229211398 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚683 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚713664212490752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111745472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚713664263693824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚788524637060608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚730276939283456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229237770 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884787894041574e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45235987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12054752691848 trainer/global_step‚4949 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105051045 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12051144028956 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788524663452160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12072325714432 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124160359 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚781588644012520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105051045 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124160359 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124186725 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45236665 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552–R¼É2è- _wandbruntime‚8996  _runtime‚8996>Jã—WéÑ _runtime‚8996.161938117 2loss/SRqqt3/assignment_loss‚1.6013455591870076 2loss/SRqqt4/assignment_loss‚1.9316002968195323 0loss/FBt2/assignment_loss‚0.1869397809289605 1loss/FBt4/assignment_loss‚0.24607120276381134 1loss/SRqqt2/assignment_loss‚1.901897320151329 2loss/FBt1/assignment_loss‚0.020769094766971346 .loss/FRt1/detection_loss‚1.621503403852421 /loss/FBt1/detection_loss‚2.9383656978607178 epoch‚4 _step‚407 0loss/FRt1/assignment_loss‚1.8803392748037975 /loss/FRt3/assignment_loss‚4.925969942756321 2loss/SRqqt1/assignment_loss‚1.2510326277936668 -loss/FBt4/detection_loss‚2.65905898809433 0loss/SRqqt2/detection_loss‚2.776302444934845 /loss/FBt2/detection_loss‚0.5733938541811621 $loss/total_loss‚47.2925820287695 -loss/FBt3/detection_loss‚2.03540151193738 .loss/FRt4/assignment_loss‚4.22089646801804 .loss/FRt4/detection_loss‚2.640558123588562 1loss/SRqqt3/detection_loss‚2.5281043648719788 % _timestamp‚1.7884787894066143e+09 trainer/global_step‚4949 /loss/FRt2/detection_loss‚1.9291132981112848 0loss/FRt2/assignment_loss‚2.7614259446660676 1loss/FBt3/assignment_loss‚0.28774846241458363 /loss/FRt3/detection_loss‚3.0260765319285183 1loss/SRqqt1/detection_loss‚0.9424617444356419 1loss/SRqqt4/detection_loss‚2.4062060899025686ù–&ç\êÖ _step‚407 0loss/FRt1/assignment_loss‚1.8803392748037975 /loss/FRt3/assignment_loss‚4.925969942756321 2loss/SRqqt1/assignment_loss‚1.2510326277936668 -loss/FBt4/detection_loss‚2.65905898809433 0loss/SRqqt2/detection_loss‚2.776302444934845 /loss/FBt2/detection_loss‚0.5733938541811621 $loss/total_loss‚47.2925820287695 -loss/FBt3/detection_loss‚2.03540151193738 .loss/FRt4/assignment_loss‚4.22089646801804 .loss/FRt4/detection_loss‚2.640558123588562 1loss/SRqqt3/detection_loss‚2.5281043648719788 % _timestamp‚1.7884787894066143e+09 trainer/global_step‚4949 /loss/FRt2/detection_loss‚1.9291132981112848 0loss/FRt2/assignment_loss‚2.7614259446660676 1loss/FBt3/assignment_loss‚0.28774846241458363 /loss/FRt3/detection_loss‚3.0260765319285183 1loss/SRqqt1/detection_loss‚0.9424617444356419 1loss/SRqqt4/detection_loss‚2.4062060899025686 _runtime‚8996.161938117 2loss/SRqqt3/assignment_loss‚1.6013455591870076 2loss/SRqqt4/assignment_loss‚1.9316002968195323 0loss/FBt2/assignment_loss‚0.1869397809289605 1loss/FBt4/assignment_loss‚0.24607120276381134 1loss/SRqqt2/assignment_loss‚1.901897320151329 2loss/FBt1/assignment_loss‚0.020769094766971346 .loss/FRt1/detection_loss‚1.621503403852421 /loss/FBt1/detection_loss‚2.9383656978607178 epoch‚4—M|–ë‚(:‹ ÉŠèÔ£’z proc.cpu.threads‚17$ disk.nvme0c0n1.in‚22900.91015625' disk.nvme0c0n1.out‚219444.337890625" disk.dm-0.out‚219444.337890625 disk.sda3.out‚ 273.3671875% disk.nvme0n1.out‚219444.337890625 network.sent‚ 1.3916438e+07$ disk./.usageGB‚1162.893398284912 disk.sda2.out‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 10.2578125# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 network.recv‚ 2.845301e+06 disk.sda1.in‚0 disk.dm-0.in‚22900.91015625 disk.dm-1.out‚ 273.3671875 disk.sda.in‚ 0.00390625 disk.sda.out‚ 273.3671875 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0) disk./.usagePercent‚78.04275169145201 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.drbd1260.out‚ 1.25390625" disk.nvme0n1.in‚22900.91015625$ memory_percent‚99.93739128112793 cpu‚25.21535130303241ì {8ì‚(:… ÉŠèÔÝÜÃz gpu.0.gpu‚64.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚183.78' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ŽŸKí‚(:… ØŠèÔ±íÇy gpu.0.gpu‚4.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÐFµ î‚(:• ØŠèÔ¶Ô½{) disk./.usagePercent‚78.04110837908974$ disk./.usageGB‚1162.868911743164" disk.nvme0n1.in‚22900.91015625 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 273.91796875 disk.sda2.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚22900.91015625) disk.nvme0c0n1.out‚219597.87353515625 disk.dm-1.out‚ 273.91796875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 273.91796875$ memory_percent‚99.93441104888916$ disk.dm-0.out‚219597.87353515625 network.sent‚ 1.4024039e+07 network.recv‚ 2.850544e+06' disk.nvme0n1.out‚219597.87353515625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.2578125# proc.memory.rssMB‚ 3274.73046875 proc.cpu.threads‚17 disk.dm-0.in‚22900.91015625' proc.memory.availableMB‚ 10.74609375( proc.memory.percent‚19.9873685836792 cpu‚25.215345808922205àIø-›ï‚(: çŠèÔí–Üx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.415399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ú—ð‚(:Œ çŠèÔy proc.cpu.threads‚17 disk.dm-1.out‚ 274.41796875 disk.sda.in‚ 0.00390625 network.sent‚ 1.4030349e+07% disk./.usageGB‚1162.8923606872559 disk.sda1.in‚0 disk.dm-0.in‚ 23028.296875 disk.sda3.in‚ 0.00390625' disk.nvme0n1.out‚219977.36572265625 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 23028.296875 disk.sda2.in‚0$ memory_percent‚99.93443489074707 cpu‚25.2154429755918( disk./.usagePercent‚78.0426820574027 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 23028.296875) disk.nvme0c0n1.out‚219977.36572265625 disk.sda3.out‚ 274.41796875 disk.drbd1260.out‚ 1.2578125 disk.sda2.out‚0& proc.memory.availableMB‚ 10.7421875 network.recv‚ 2.855366e+06$ disk.dm-0.out‚219977.36572265625 disk.sda.out‚ 274.41796875# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.98736858367924׈›ñ‚(: öŠèÔ•Ÿóx disk.sda2.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 10.73828125# proc.memory.rssMB‚ 3274.73046875) disk./.usagePercent‚78.05106681616414 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.26171875 disk.nvme0n1.in‚ 23028.296875$ memory_percent‚99.93445873260498( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17 network.sent‚ 1.4035221e+07 disk.sda3.in‚ 0.00390625 disk.sda2.out‚0 cpu‚25.21550169041699 network.recv‚ 2.859004e+06% disk./.usageGB‚1163.0172996520996) disk.nvme0c0n1.out‚220300.22021484375 disk.dm-0.in‚ 23028.296875 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 274.86328125 disk.sda3.out‚ 274.86328125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 23028.296875$ disk.dm-0.out‚220300.22021484375 disk.dm-1.out‚ 274.86328125' disk.nvme0n1.out‚220300.22021484375¿O1òŒò‚(: öŠèÔêì¼y gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.3' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.86 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;!¾ó‚(:’ …‹èÔ뜧w gpu.0.gpu‚95.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚142.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.493599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É$°›ô‚(: …‹èÔÑÏñw' disk.nvme0n1.out‚220669.89599609375 disk.sda2.in‚0 disk.sda1.in‚0( disk./.usagePercent‚77.9946115226984 disk.sda3.out‚ 275.23046875 disk.nvme0n1.in‚ 23028.296875 disk.sda1.out‚0$ memory_percent‚99.93445873260498# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 cpu‚25.215509301057487" disk.nvme0c0n1.in‚ 23028.296875) disk.nvme0c0n1.out‚220669.89599609375' proc.memory.availableMB‚ 10.73828125 proc.cpu.threads‚17 network.recv‚ 2.863686e+06% disk./.usageGB‚1162.1760749816895 disk.dm-0.in‚ 23028.296875 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 275.23046875 disk.sda.in‚ 0.00390625 disk.drbd1260.out‚ 1.26171875 disk.sda2.out‚0 network.sent‚ 1.4041426e+07$ disk.dm-0.out‚220669.89599609375 disk.sda.out‚ 275.23046875 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0舒õ‚(:‡ ”‹èÔ‘å‰x gpu.0.gpu‚49.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚272.477' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4954 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Sà‘ö‚(:† ”‹èÔü÷êx disk.sda3.out‚ 275.7109375 disk.sda2.in‚0 disk.dm-0.in‚ 23028.296875 disk.dm-1.out‚ 275.7109375 disk.sda.in‚ 0.00390625 disk.sda.out‚ 275.7109375 proc.cpu.threads‚17% disk./.usageGB‚1162.1755599975586 disk.sda3.in‚ 0.00390625 disk.drbd1260.out‚1.265625 disk.nvme0n1.in‚ 23028.296875% disk.nvme0n1.out‚220944.544921875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 23028.296875 disk.dm-1.in‚ 0.00390625 network.recv‚ 2.867131e+06 disk.sda2.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚220944.544921875$ memory_percent‚99.93326663970947# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 cpu‚25.215533059254867 network.sent‚ 1.4046397e+07 disk.drbd1260.in‚0" disk.dm-0.out‚220944.544921875' proc.memory.availableMB‚ 10.93359375) disk./.usagePercent‚77.99457696168129KãúÊ2÷- _wandbruntime‚9084  _runtime‚9084>Dx4ÆøÀ & lr-AdamW/pg2‚0.0004754298653072755 trainer/global_step‚4999 $ _timestamp‚1.788478877690291e+09 _runtime‚9084.447423127 _step‚408 & lr-AdamW/pg1‚0.0004754298653072755ø¨Í»ËùÅ & lr-AdamW/pg2‚0.0004754298653072755 trainer/global_step‚4999 $ _timestamp‚1.788478877690291e+09 _runtime‚9084.447423127 _step‚408 & lr-AdamW/pg1‚0.0004754298653072755˜É¶´2ú- _wandbruntime‚9084  _runtime‚9084›T¦oûŒP H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚776900024136336 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884788776944141e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125367494 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12194064390656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚796059763845120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 _runtime‚9084.452404585 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12190453729792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚739 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67136930 # TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚776899998708368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231419309 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚796059737453568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚808253828235776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45670456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12194064390656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231445655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16776198305792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚789069065189546 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106051815 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16776124270592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚409 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125367494 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45669722 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚796059763845120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112807391 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚808250191183360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚734 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚789072699280198 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106051821 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106051821 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚720489492754432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚808253828235776 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125393834 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚796059737453568 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚4999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125393834 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚737265691060224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚808250191183360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720489441551360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12194064390656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231445655 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16776198305792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚789069065189546 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106051815 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16776124270592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚409 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125367494 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45669722 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚796059763845120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112807391 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚808250191183360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚734 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚789072699280198 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106051821 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106051821 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚720489492754432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚808253828235776 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125393834 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚796059737453568 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚4999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125393834 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚737265691060224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚808250191183360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚720489441551360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚776900024136336 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884788776944141e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125367494 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12194064390656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚796059763845120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 _runtime‚9084.452404585 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12190453729792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚739 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67136930 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448™’ÁŠ2ý- _wandbruntime‚9086  _runtime‚9086y@è'þ–N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106072239 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125392061 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67149735 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12192864211456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚808410016289792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45679255 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12175081447356 % _timestamp‚1.7884788796349998e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚796213541404160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231490672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12196474885632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67149740 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚789225434824828 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231464300 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106072239 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112828292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737408188793344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024¼GéÜu ÿ›N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125418427 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚808406379224064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106072245 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125418427 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚737408314018304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16779456852992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12196474885632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚796213515012608 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231490672 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12196474885632 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67149740 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚789225434824828 UHDeviceStatsMonitor.on_train_batch_end/requested_a¹Vq¬bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231464300 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106072239 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112828292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737408188793344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106072239 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125392061 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67149735 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12192864211456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚808410016289792 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45679255 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12175081447356 % _timestamp‚1.7884788796349998e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚796213541404160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚803748615405568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚744396485808128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚816063244684800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233698557 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107073021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚816059607632384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚796709490092506 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126599196 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803748641797120 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107073015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46113458 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16942598842368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784416460765168 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126625536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46114199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126599196 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67777180 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233672211 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803748615405568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12314602887680 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚727454012204032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5049 _step‚413 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12314602887680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12289395236686 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107073015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12310992226816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚796705856001854 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884789721509068e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803748641797120 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113891384 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 _runtime‚ 9178.90682163 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67777185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784416486193136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚744396611046400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚727453961000960 ,í”P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚727453961000960 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126625536 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16942524807168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚803748615405568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚744396485808128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚816063244684800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233698557 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107073021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚816059607632384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚796709490092506 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126599196 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803748641797120 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107073015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46113458 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16942598842368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚784416460765168 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126625536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46114199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126599196 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67777180 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233672211 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803748615405568 MGDeviceStatsMonitor.on_trã8 )ain_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12314602887680 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚727454012204032 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5049 _step‚413 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12314602887680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12289395236686 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107073015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12310992226816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚796705856001854 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884789721509068e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803748641797120 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113891384 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 _runtime‚ 9178.90682163 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67777185 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784416486193136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚744396611046400 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532eIdEœ•‚(:‘ ý‹èÔÝùx$ disk.nvme0c0n1.in‚23296.78515625 disk.dm-1.in‚ 0.00390625 network.sent‚ 1.4189114e+07 disk.drbd1260.out‚ 1.27734375" disk.nvme0n1.in‚23296.78515625 disk.sda1.in‚0( disk.nvme0c0n1.out‚223189.9365234375 disk.dm-0.in‚23296.78515625 disk.dm-1.out‚ 278.72265625 disk.sda.out‚ 278.72265625) disk./.usagePercent‚78.01465435255407% disk./.usageGB‚1162.4747276306152 disk.drbd1260.in‚0# disk.dm-0.out‚223189.9365234375 disk.sda.in‚ 0.00390625( proc.memory.percent‚19.9873685836792 cpu‚25.215814819055836 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 278.72265625& disk.nvme0n1.out‚223189.9365234375 disk.sda2.out‚0# memory_percent‚99.9354362487793% proc.memory.availableMB‚ 10.578125# proc.memory.rssMB‚ 3274.73046875 network.recv‚ 2.898782e+06 disk.sda2.in‚0Õ»Eþ’–‚(:‡ ý‹èÔí·ýy gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚204.079' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.8158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ëCM2—- _wandbruntime‚9180  _runtime‚9180Ɔº'˜–N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113912291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67789985 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚744539192334336 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233743574 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 % _timestamp‚1.7884789736526945e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚9180.414196071 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107093439 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚727593251389952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233717202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚803902419356160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12313402708480 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12317013382656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚796862225637136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚803902392964608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚816219432738816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107093445 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67789990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16945815719424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233743574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚816215795673088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107093445 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107093439 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727593302593024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16945889741312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233717202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46122306 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784566790006304 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 trainer/global_step‚5049 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚744539067109376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113912291 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67789985 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚744539192334336 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233743574 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 % _timestamp‚1.7884789736526945e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚9180.414196071 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107093439 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚727593251389952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233717202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚803902419356160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12313402708480 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12317013382656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚796862225637136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚803902392964608 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚816219432738816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107093445 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67789990 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16945815719424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233743574 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚816215795673088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107093445 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107093439 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727593302593024 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16945889741312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712ž.32š- _wandbruntime‚9180  _runtime‚9180Æš„øW›Ñ .loss/FRt3/assignment_loss‚4.51886719923753 /loss/FRt1/detection_loss‚1.7070617998439073 .loss/FBt2/detection_loss‚0.785224477787103 _runtime‚9180.418296411 /loss/FRt4/assignment_loss‚3.530077244226749 1loss/SRqqt1/assignment_loss‚1.170849042491281 2loss/SRqqt3/assignment_loss‚1.2218663480419378 %loss/total_loss‚45.56370456719044 epoch‚4 2loss/SRqqt4/assignment_loss‚1.8864492682947054 .loss/FRt4/detection_loss‚3.129170814385781 1loss/SRqqt1/detection_loss‚1.0198575352965775 1loss/SRqqt2/detection_loss‚2.6279736820020174 .loss/FBt4/detection_loss‚2.299967437982559 trainer/global_step‚5049 _step‚415 /loss/FRt1/assignment_loss‚2.091785335377471 2loss/SRqqt2/assignment_loss‚1.5409979686925286 1loss/FBt2/assignment_loss‚0.11501315892673937 0loss/FBt3/assignment_loss‚0.4069579399522179 /loss/FRt2/detection_loss‚1.7194719773436349 1loss/SRqqt4/detection_loss‚2.6767297663070537 0loss/FRt2/assignment_loss‚2.4521756304634943 % _timestamp‚1.7884789736613495e+09 .loss/FBt1/detection_loss‚2.473302401602268 /loss/FBt3/detection_loss‚2.3968505951074452 1loss/FBt1/assignment_loss‚0.08420432719140081 /loss/FBt4/assignment_loss‚0.290314723856032 /loss/FRt3/detection_loss‚3.0128615406843333 1loss/SRqqt3/detection_loss‚2.4056743520956774 ç¥Æ\œÖ 0loss/FBt3/assignment_loss‚0.4069579399522179 /loss/FRt2/detection_loss‚1.7194719773436349 1loss/SRqqt4/detection_loss‚2.6767297663070537 0loss/FRt2/assignment_loss‚2.4521756304634943 % _timestamp‚1.7884789736613495e+09 .loss/FBt1/detection_loss‚2.473302401602268 /loss/FBt3/detection_loss‚2.3968505951074452 1loss/FBt1/assignment_loss‚0.08420432719140081 /loss/FBt4/assignment_loss‚0.290314723856032 /loss/FRt3/detection_loss‚3.0128615406843333 1loss/SRqqt3/detection_loss‚2.4056743520956774 .loss/FRt3/assignment_loss‚4.51886719923753 /loss/FRt1/detection_loss‚1.7070617998439073 .loss/FBt2/detection_loss‚0.785224477787103 _runtime‚9180.418296411 /loss/FRt4/assignment_loss‚3.530077244226749 1loss/SRqqt1/assignment_loss‚1.170849042491281 2loss/SRqqt3/assignment_loss‚1.2218663480419378 %loss/total_loss‚45.56370456719044 epoch‚4 2loss/SRqqt4/assignment_loss‚1.8864492682947054 .loss/FRt4/detection_loss‚3.129170814385781 1loss/SRqqt1/detection_loss‚1.0198575352965775 1loss/SRqqt2/detection_loss‚2.6279736820020174 .loss/FBt4/detection_loss‚2.299967437982559 trainer/global_step‚5049 _step‚415 /loss/FRt1/assignment_loss‚2.091785335377471 2loss/SRqqt2/assignment_loss‚1.5409979686925286 1loss/FBt2/assignment_loss‚0.11501315892673937ŸÖ ë‚(:… ŒŒèÔ‡ÝÖx% disk./.usageGB‚1162.4726295471191 disk.sda.out‚ 279.1171875 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0# disk.nvme0n1.out‚223561.3046875 disk.sda2.out‚0( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17 network.sent‚ 1.4285232e+07 disk.dm-0.in‚23296.78515625 disk.dm-0.out‚223561.3046875 disk.dm-1.in‚ 0.00390625 disk.drbd1260.out‚1.28125 cpu‚25.215952203327607 network.recv‚ 2.904005e+06) disk./.usagePercent‚78.01451354841025$ disk.nvme0c0n1.in‚23296.78515625% disk.nvme0c0n1.out‚223561.3046875 disk.sda3.out‚ 279.1171875 disk.sda2.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 10.51953125 disk.dm-1.out‚ 279.1171875 disk.sda.in‚ 0.00390625" disk.nvme0n1.in‚23296.78515625 disk.sda1.in‚0$ memory_percent‚99.93579387664795# proc.memory.rssMB‚ 3274.73046875›” Ÿ›ž‚(: ŒŒèÔ¥”€z gpu.0.gpu‚28.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.61' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.522000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À»Ÿ‚(:’ ›ŒèÔ¢¢§y gpu.0.gpu‚54.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚213.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©£k#‹ ‚(:€ ›ŒèÔˆ×Üy# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 disk.sda1.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.93433952331543 proc.cpu.threads‚17 network.sent‚ 1.4383632e+07 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 23355.78125 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 279.62890625 disk.drbd1260.out‚1.28125 disk.sda2.in‚0 cpu‚25.216028199116778 network.recv‚ 2.910685e+06( disk./.usagePercent‚78.0145109883349% disk./.usageGB‚1162.4725914001465& disk.nvme0c0n1.out‚223677.41015625! disk.dm-0.out‚223677.41015625 disk.dm-1.out‚ 279.62890625 disk.sda.in‚ 0.00390625 disk.nvme0n1.in‚ 23355.78125 disk.dm-0.in‚ 23355.78125 disk.sda.out‚ 279.62890625 disk.sda3.in‚ 0.00390625$ disk.nvme0n1.out‚223677.41015625 disk.sda2.out‚0& proc.memory.availableMB‚ 10.7578125ã)N ¡‚(:† ªŒèÔ¹ð›y gpu.0.gpu‚28.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu¸%úñ.0.temp‚37.0 gpu.0.powerWatts‚162.471' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04J¶š¢‚(: ªŒèÔºÿçz disk.sda3.in‚ 0.00390625 disk.sda2.out‚0 disk.sda1.in‚0( disk./.usagePercent‚77.9919751571108$ disk./.usageGB‚1162.136791229248 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.28515625 disk.sda1.out‚0$ memory_percent‚99.93588924407959( proc.memory.percent‚19.9873685836792 disk.dm-1.out‚ 280.09765625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 280.09765625 disk.nvme0n1.in‚ 23406.921875# proc.memory.rssMB‚ 3274.73046875 cpu‚25.215808443746468 proc.cpu.threads‚17 network.sent‚ 1.4388573e+07" disk.nvme0c0n1.in‚ 23406.921875) disk.nvme0c0n1.out‚224072.54443359375 disk.dm-0.in‚ 23406.921875 disk.sda3.out‚ 280.09765625' disk.nvme0n1.out‚224072.54443359375 disk.sda2.in‚0' proc.memory.availableMB‚ 10.50390625 network.recv‚ 2.914116e+06$ disk.dm-0.out‚224072.544433593754 ’*’£‚(:‡ ¹ŒèÔ™™“w gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚245.828' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.1656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H>ÄÇ”¤‚(:‰ ¹ŒèÔÙÊz% disk./.usageGB‚1162.6132698059082 disk.nvme0n1.in‚ 23406.921875 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 280.421875 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 network.recv‚ 2.918213e+06 disk.sda2.out‚0 disk.dm-0.in‚ 23406.921875$ disk.dm-0.out‚224587.42724609375 disk.sda.out‚ 280.421875 disk.sda3.out‚ 280.421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.28515625 network.sent‚ 1.4394844e+07) disk.nvme0c0n1.out‚224587.42724609375 disk.dm-1.in‚ 0.00390625& proc.memory.availableMB‚ 10.4609375# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792) disk./.usagePercent‚78.02395203418261' disk.nvme0n1.out‚224587.42724609375" disk.nvme0c0n1.in‚ 23406.921875# memory_percent‚99.9361515045166 cpu‚25.215596888409657 proc.cpu.threads‚17š’¥‚(:‡ ÈŒèÔ‚üÍx gpu.0.gpu‚72.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚181.567' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ׯà&•¦‚(:Š ÈŒèÔùò—y proc.cpu.threads‚17) disk./.usagePercent‚78.02546785479285( disk.nvme0c0n1.out‚224909.2021484375 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 23406.921875 disk.sda2.in‚0# proc.memory.rssMB‚ 3274.73046875 cpu‚25.21533905973395 network.sent‚ 1.4398899e+07$ disk./.usageGB‚1162.635856628418 disk.dm-0.in‚ 23406.921875 disk.drbd1260.out‚ 1.2890625 disk.sda2.out‚0' proc.memory.availableMB‚ 10.51171875 disk.dm-1.out‚ 280.84765625 disk.sda.in‚ 0.00390625& disk.nvme0n1.out‚224909.2021484375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.93584156036377( proc.memory.percent‚19.9873685836792 disk.sda3.out‚ 280.84765625 network.recv‚ 2.92125e+06" disk.nvme0c0n1.in‚ 23406.921875# disk.dm-0.out‚224909.2021484375 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 280.84765625(0犖§‚(:‹ ׌èÔ„½Èx disk.sda2.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚78.0238928964422( disk.nvme0c0n1.out‚225351.8740234375 disk.dm-0.in‚ 23406.921875 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 23406.921875 disk.sda1.in‚0# proc.memory.rssMB‚ 3274.73046875% disk./.usageGB‚1162.6123886108398" disk.nvme0c0n1.in‚ 23406.921875 disk.dm-1.out‚ 281.40234375 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 281.40234375 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚99.93593692779541 network.sent‚ 1.4407364e+07 disk.sda.out‚ 281.40234375 disk.drbd1260.out‚ 1.2890625& disk.nvme0n1.out‚225351.8740234375' proc.memory.availableMB‚ 10.49609375( proc.memory.percent‚19.9873685836792 cpu‚25.21518458341046 network.recv‚ 2.925893e+06# disk.dm-0.out‚225351.8740234375 disk.sda.in‚ 0.00390625’å¬ë‘¨‚(:† ׌èÔÑÔ±y gpu.0.gpu‚21.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô®t 2©- _wandbruntime‚9275  _runtime‚9275Tw›Ôɪà ' lr-AdamW/pg1‚0.00046700689488857097 ' lr-AdamW/pg2‚0.00046700689488857097 trainer/global_step‚5099 % _timestamp‚1.7884790691960785e+09 _runtime‚9275.951741513 _step‚416«‰?Î«È _runtime‚9275.951741513 _step‚416 ' lr-AdamW/pg1‚0.00046700689488857097 ' lr-AdamW/pg2‚0.00046700689488857097 trainer/global_step‚5099 % _timestamp‚1.7884790691960785e+09 h7ß2¬- _wandbruntime‚9275  _runtime‚9275WÚ°8(­‹P WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚823872661133824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114974397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚734418531653632 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108094215 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127857238 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12435141384704 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127857238 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12409723992194 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚804346280904814 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚791932922821968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12431530723840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811437519749120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235951459 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚791932948249936 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811437493357568 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108094215 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚823869024081408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17108874849280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚751527478440960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚734418480450560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108094221 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚823872661133824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12413332654878 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚804342646814162 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68417435 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127830898 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12431530723840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46557687 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17108946787328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 trainer/global_step‚5099 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚823869024081408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12435141384704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68417430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 _step‚417 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811437493357568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127830898 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46556967 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚751527355299840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚725 $ _timestamp‚1.788479069197538e+09 _runtime‚9275.953327815 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249Iø€(®P MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 _step‚417 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚811437493357568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127830898 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46556967 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚751527355299840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚725 $ _timestamp‚1.788479069197538e+09 _runtime‚9275.953327815 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚823872661133824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114974397 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚734418531653632 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108094215 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127857238 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12435141384704 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127857238 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12409723992194 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚804346280904814 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚791932922821968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12431530723840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811437519749120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235951459 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚791932948249936 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811437493357568 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108094215 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚823869024081408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17108874849280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚751527478440960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚734418480450560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108094221 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚823872661133824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12413332654878 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚804342646814162 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68417435 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127830898 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12431530723840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46557687 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17108946787328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 trainer/global_step‚5099 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚823869024081408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12435141384704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68417430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚720 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684¡l\æ2¯- _wandbruntime‚9276  _runtime‚9276˜zÓ­=°“N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17112268848128 _runtime‚ 9276.99727385 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17112196923392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚824025212122112 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108114645 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12412130295480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 _step‚418 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚824028849187840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235996476 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108114645 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127881831 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811591297308160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108114639 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚811591297308160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚804499016449444 # _timestamp‚1.78847907024109e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12437551879680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127881831 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114996136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46566589 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚734557770839552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚792083252063104 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108114639 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235970104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235996476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811591270916608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46565901 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚792083277491072 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚751669967762944 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12437551879680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12433941205504 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114996829 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12433941205504 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚824028849187840 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 trainer/global_step‚5099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127855465 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127855465 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68430240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68430235 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688• ”å'±˜N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46565901 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚792083277491072 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚751669967762944 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12437551879680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12433941205504 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114996829 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12433941205504 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚824028849187840 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 trainer/global_step‚5099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127855465 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127855465 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68430240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68430235 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17112268848128 _runtime‚ 9276.99727385 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17112196923392 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚824025212122112 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108114645 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12412130295480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 _step‚418 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚824028849187840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235996476 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108114645 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127881831 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811591297308160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108114639 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚811591297308160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚804499016449444 # _timestamp‚1.78847907024109e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12437551879680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127881831 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114996136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46566589 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚734557770839552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚792083252063104 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108114639 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235970104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235996476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811591270916608¢“tå2²- _wandbruntime‚9276  _runtime‚9276ù}MçZ³Ô /loss/FRt2/assignment_loss‚2.297865363182845 1loss/SRqqt3/assignment_loss‚1.688546720147133 1loss/SRqqt1/detection_loss‚1.1296701638270978 1loss/SRqqt4/detection_loss‚2.0167550866802535 %loss/total_loss‚46.25744928533038 % _timestamp‚1.7884790702438598e+09 /loss/FRt4/assignment_loss‚3.789827462954399 epoch‚4 trainer/global_step‚5099 _step‚419 /loss/FRt1/assignment_loss‚1.650087685244424 2loss/SRqqt1/assignment_loss‚1.1447738568743935 2loss/SRqqt4/assignment_loss‚2.1805903447998896 -loss/FBt3/detection_loss‚2.16029558579127 2loss/FBt1/assignment_loss‚0.017753400427556205 0loss/FBt3/assignment_loss‚0.2097984205341033 1loss/FBt2/assignment_loss‚0.28073003475913977 1loss/FBt4/assignment_loss‚0.28580080870709096 /loss/FRt1/detection_loss‚1.5489311173649964 .loss/FRt4/detection_loss‚2.620572629647377 1loss/SRqqt3/detection_loss‚2.8944799810647965 .loss/FBt1/detection_loss‚2.403960386912028 _runtime‚9276.999318486 2loss/SRqqt2/assignment_loss‚2.0252490760315034 /loss/FRt2/detection_loss‚1.8307951066505026 /loss/FRt3/detection_loss‚3.3783392906188965 1loss/SRqqt2/detection_loss‚2.4683365665730976 /loss/FBt2/detection_loss‚0.8242052756814342 /loss/FRt3/assignment_loss‚4.633029492277848 /loss/FBt4/detection_loss‚2.7770554285783033„YR_´Ù 2loss/SRqqt2/assignment_loss‚2.0252490760315034 /loss/FRt2/detection_loss‚1.8307951066505026 /loss/FRt3/detection_loss‚3.3783392906188965 1loss/SRqqt2/detection_loss‚2.4683365665730976 /loss/FBt2/detection_loss‚0.8242052756814342 /loss/FRt3/assignment_loss‚4.633029492277848 /loss/FBt4/detection_loss‚2.7770554285783033 /loss/FRt2/assignment_loss‚2.297865363182845 1loss/SRqqt3/assignment_loss‚1.688546720147133 1loss/SRqqt1/detection_loss‚1.1296701638270978 1loss/SRqqt4/detection_loss‚2.0167550866802535 %loss/total_loss‚46.25744928533038 % _timestamp‚1.7884790702438598e+09 /loss/FRt4/assignment_loss‚3.789827462954399 epoch‚4 trainer/global_step‚5099 _step‚419 /loss/FRt1/assignment_loss‚1.650087685244424 2loss/SRqqt1/assignment_loss‚1.1447738568743935 2loss/SRqqt4/assignment_loss‚2.1805903447998896 -loss/FBt3/detection_loss‚2.16029558579127 2loss/FBt1/assignment_loss‚0.017753400427556205 0loss/FBt3/assignment_loss‚0.2097984205341033 1loss/FBt2/assignment_loss‚0.28073003475913977 1loss/FBt4/assignment_loss‚0.28580080870709096 /loss/FRt1/detection_loss‚1.5489311173649964 .loss/FRt4/detection_loss‚2.620572629647377 1loss/SRqqt3/detection_loss‚2.8944799810647965 .loss/FBt1/detection_loss‚2.403960386912028 _runtime‚9276.999318486£ZðGš µ‚(:• æŒèÔ˜ùØx network.sent‚ 1.4410286e+07( disk./.usagePercent‚78.0255482411586% disk./.usageGB‚1162.6370544433594 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 network.recv‚ 2.929085e+06 disk.sda1.out‚0$ disk.nvme0c0n1.in‚23560.35546875) disk.nvme0c0n1.out‚225683.46240234375 disk.dm-0.in‚23560.35546875$ memory_percent‚99.93634223937988 proc.cpu.threads‚17 disk.sda3.out‚ 281.80859375& proc.memory.availableMB‚ 10.4296875# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 cpu‚25.214919641456156" disk.nvme0n1.in‚23560.35546875' disk.nvme0n1.out‚225683.46240234375$ disk.dm-0.out‚225683.46240234375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 281.80859375 disk.sda.out‚ 281.80859375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.29296875##š¶‚(: æŒèÔŽê®y gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.772000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ͮƚ·‚(: õŒèÔÐÍ„y disk.sda.in‚ 0.00390625' proc.memory.availableMB‚ 10.16796875 cpu‚25.214681100548265% disk./.usageGB‚1162.6370544433594' disk.nvme0n1.out‚225933.89208984375 disk.sda1.in‚0 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 network.recv‚ 2.935661e+06( disk./.usagePercent‚78.0255482411586 disk.sda2.in‚0$ disk.nvme0c0n1.in‚23560.35546875 disk.dm-0.in‚23560.35546875$ disk.dm-0.out‚225933.89208984375 disk.sda.out‚ 282.109375# proc.memory.rssMB‚ 3274.73046875 network.sent‚ 1.451913e+07 disk.sda2.out‚0) disk.nvme0c0n1.out‚225933.89208984375 disk.sda3.out‚ 282.109375 disk.drbd1260.out‚ 1.29296875$ memory_percent‚99.93793964385986( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17" disk.nvme0n1.in‚23560.35546875 disk.sda1.out‚0 disk.dm-1.out‚ 282.109375g–h‘¸‚(:† õŒèÔ£Ñæy gpu.0.gpu‚52.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚204.131' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.8262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Û5l™¹‚(:Ž „èÔ¼×øw disk.sda.out‚ 282.7734375 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3274.73046875% disk./.usageGB‚1161.9861640930176( disk.nvme0c0n1.out‚226159.6552734375 disk.drbd1260.out‚1.296875$ memory_percent‚99.93641376495361 cpu‚25.214451278962706 network.recv‚ 2.939127e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚23560.35546875 disk.dm-0.in‚23560.35546875 disk.dm-1.out‚ 282.7734375 disk.sda3.out‚ 282.7734375 network.sent‚ 1.4524163e+07( disk./.usagePercent‚77.9818664436138# disk.dm-0.out‚226159.6552734375 disk.sda.in‚ 0.00390625' proc.memory.availableMB‚ 10.41796875( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17" disk.nvme0n1.in‚23560.35546875& disk.nvme0n1.out‚226159.6552734375 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625!þ’º‚(:‡ „èÔÓ…–x gpu.0.gpu‚40.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚273.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nû‚(:’ “èÔ½†ây gpu.0.gpu‚62.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚115.416' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.083199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ÒÎ)š¼‚(: “èÔÒ¦Óz disk.sda3.out‚ 283.3515625( disk.nvme0c0n1.out‚226618.9326171875$ memory_percent‚99.93641376495361( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17 network.sent‚ 1.4530477e+07 network.recv‚ 2.944098e+06) disk./.usagePercent‚77.98340863300005% disk./.usageGB‚1162.0091438293457 disk.dm-0.in‚23560.35546875# disk.dm-0.out‚226618.9326171875 disk.dm-1.out‚ 283.3515625" disk.nvme0n1.in‚23560.35546875 disk.sda.in‚ 0.00390625 disk.sda.out‚ 283.3515625 disk.sda2.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 10.41796875# proc.memory.rssMB‚ 3274.73046875 cpu‚25.214201084825262 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚1.296875 disk.dm-1.in‚ 0.00390625& disk.nvme0n1.out‚226618.9326171875 disk.sda2.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚23560.355468751#¡C2½- _wandbruntime‚9343  _runtime‚9343_ü¢>ɾà ' lr-AdamW/pg2‚0.00046275086771290294 trainer/global_step‚5149 % _timestamp‚1.7884791363568923e+09 _runtime‚9343.113714993 _step‚420 ' lr-AdamW/pg1‚0.00046275086771290294ᮞ-Î¿È ' lr-AdamW/pg1‚0.00046275086771290294 ' lr-AdamW/pg2‚0.00046275086771290294 trainer/global_step‚5149 % _timestamp‚1.7884791363568923e+09 _runtime‚9343.113714993 _step‚420¤BÎ0@2À- _wandbruntime‚9343  _runtime‚9343Q‰5¹(ÁŠP YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819126371309568 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12555679881728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799449410306736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12552069220864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚811979437626470 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚831682077582848 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109115421 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17275362195456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _step‚421 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129088940 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12552069220864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 trainer/global_step‚5149 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109115415 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚799449384878768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109115421 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116056687 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚831678440530432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238204361 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚758658362095616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109115415 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 _runtime‚ 9343.1144558 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129088940 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚758658487333888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819126397701120 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129062600 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129062600 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116055952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69057680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚831678440530432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12552069220864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚811979437626470 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚831682077582848 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109115421 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17275362195456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 _step‚421 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129088940 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12552069220864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 trainer/global_step‚5149 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109115415 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚799449384878768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109115421 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116056687 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚831678440530432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238204361 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚758658362095616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109115415 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚730 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 _runtime‚ 9343.1144558 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129088940 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚758658487333888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¥Š‰u2Ã- _wandbruntime‚9344  _runtime‚9344ϲ¶¯'Ä–N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69070490 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚758800989235712 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17278770871296 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238223006 _runtime‚9344.279586979 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12558090376704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47007132 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚741522341492224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚799599714119904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109135839 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819280148868608 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238223006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116078295 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 trainer/global_step‚5149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 _step‚422 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238249378 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚812132173170892 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116077617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69070485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12536067713880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47007805 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12558090376704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109135839 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚741522290289152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚799599739547872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129113533 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚831838265636864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17278698946560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12554479702528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238249378 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚812132173170892 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116077617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69070485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12536067713880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47007805 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12558090376704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109135839 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚741522290289152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚799599739547872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129113533 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚831838265636864 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17278698946560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12554479702528 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238223006 _runtime‚9344.279586979 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12558090376704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47007132 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚741522341492224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚799599714119904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109135839 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819280148868608 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238223006 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116078295 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 trainer/global_step‚5149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 _step‚422 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B‘Ñ‚(:† Þèԛʽx gpu.0.gpu‚12.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚144.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P¢’”Ò‚(:‰ ÞèÔ¼¦Üx disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 285.5234375# proc.memory.rssMB‚ 3274.73046875 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17 network.sent‚ 1.4660555e+07 disk.sda.out‚ 285.5234375 disk.sda3.out‚ 285.5234375 disk.nvme0n1.in‚ 23830.328125 disk.dm-0.in‚ 23830.328125$ proc.memory.availableMB‚10.34375 cpu‚25.21302068338696) disk./.usagePercent‚77.91466625792846 disk.sda.in‚ 0.00390625 disk.drbd1260.out‚ 1.30859375$ disk.dm-0.out‚228075.31787109375# memory_percent‚99.9368667602539 network.recv‚ 2.965373e+06% disk./.usageGB‚1160.9848327636719' disk.nvme0n1.out‚228075.31787109375" disk.nvme0c0n1.in‚ 23830.328125) disk.nvme0c0n1.out‚228075.31787109375î>Ô2Ó- _wandbruntime‚9413  _runtime‚9413´t:ÉÔà ' lr-AdamW/pg1‚0.00045846632645022626 ' lr-AdamW/pg2‚0.00045846632645022626 trainer/global_step‚5199 % _timestamp‚1.7884792071763191e+09 _runtime‚9413.932723383 _step‚424âD@)ÎÕÈ ' lr-AdamW/pg1‚0.00045846632645022626 ' lr-AdamW/pg2‚0.00045846632645022626 trainer/global_step‚5199 % _timestamp‚1.7884792071763191e+09 _runtime‚9413.932723383 _step‚424¨r§Ý2Ö- _wandbruntime‚9413  _runtime‚9413úA½(׋P LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12672607717888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12672607717888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110136615 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110136621 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240457263 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110136621 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117138512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12653990165894 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17441701894656 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚826815249261568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240430917 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806965872363536 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748347519349760 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚9413.936686866 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚826815249261568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110136615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788479207180022e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130294302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47439880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚765789344385536 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚839487856979456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚826815275653120 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12650381503210 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12676218378752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748347570552832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚425 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚839487856979456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚765789221244416 trainer/global_step‚5199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚839491494031872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚819616228438778 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69697935 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117137810iø'ÿ(ØP \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚806965846935568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚826815275653120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819619862529430 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚765789221244416 trainer/global_step‚5199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚839491494031872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚819616228438778 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69697935 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117137810 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12672607717888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12672607717888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110136615 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110136621 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240457263 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110136621 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117138512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12653990165894 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17441701894656 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚826815249261568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 240430917 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806965872363536 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748347519349760 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚9413.936686866 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚826815249261568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110136615 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788479207180022e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130294302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47439880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚765789344385536 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚839487856979456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚826815275653120 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12650381503210 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12676218378752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748347570552832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚425 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚839487856979456 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117159281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69710735 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110157045 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110157039 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17445086418432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12656396469388 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117159958 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130345235 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765931826329600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240475908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚748486860941824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12675018199552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsc•‰£Monitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240502280 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130345235 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚839644045020160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚765931947360256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47449218 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12652787806496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69827584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚819772598074060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130318869 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12678628873728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12678628873728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚807116176176704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚826969053212160 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚9415.888775382 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚839647682085888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240502280 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240475908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69710740 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121030656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚826969026820608 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚826969053212160 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚807116201604672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ð¹ìC!'Û›N B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚826969053212160 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚807116201604672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826969026820608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12675018199552 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚748486809738752 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17445016590848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110157045 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚839647682085888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117159281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69710735 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110157045 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110157039 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17445086418432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12656396469388 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117159958 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130345235 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765931826329600 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240475908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚748486860941824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12675018199552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240502280 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130345235 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚839644045020160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚765931947360256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47449218 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12652787806496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚5199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69827584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚819772598074060 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130318869 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12678628873728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12678628873728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚807116176176704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚826969053212160 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚9415.888775382 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚839647682085888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240502280 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240475908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69710740 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121030656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚826969026820608 F2ë- _wandbruntime‚9504  _runtime‚9504®ßâÇìÁ & lr-AdamW/pg1‚0.0004541541639925182 & lr-AdamW/pg2‚0.0004541541639925182 trainer/global_step‚5249 % _timestamp‚1.7884792973707626e+09 _runtime‚9504.127724103 _step‚428÷sœÌíÆ _step‚428 & lr-AdamW/pg1‚0.0004541541639925182 & lr-AdamW/pg2‚0.0004541541639925182 trainer/global_step‚5249 % _timestamp‚1.7884792973707626e+09 _runtime‚9504.127724103¬²'"á2î- _wandbruntime‚9504  _runtime‚9504-W4™ïïŒP UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111157821 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111157815 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚827256653341738 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17608305350144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12774318921402 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 % _timestamp‚1.7884792973746634e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12793146214912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚827253019251086 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚814482308992368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111157821 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚847300910480896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚834504127213568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70338185 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17608233412096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131552344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚772920395352576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131526004 \IDeviceStatsMonitor.onŒØÜÖ#_train_batch_start/active_bytes.small_pool.allocated‚12796756875776 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834504153605120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚755312038799360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚755312090002432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131552344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242683819 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111157815 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚834504127213568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131526004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12796756875776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 _runtime‚9504.132954764 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118220870 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70338180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12793146214912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47883389 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚847297273428480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚814482334420336¯B'(ð‘P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111157815 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚827256653341738 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17608305350144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12774318921402 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 % _timestamp‚1.7884792973746634e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12793146214912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚827253019251086 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚814482308992368 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111157821 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚847300910480896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚834504127213568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70338185 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17608233412096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131552344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚772920395352576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131526004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12796756875776 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834504153605120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚755312038799360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚755312090002432 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131552344 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242683819 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111157815 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚834504127213568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131526004 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12796756875776 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 _runtime‚9504.132954764 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118220870 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70338180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12793146214912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47883389 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚847297273428480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚814482334420336 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111157821­Š@Ðæ2ñ- _wandbruntime‚9506  _runtime‚9506ænK>'ò–N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884793001288166e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚834657931164160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242755182 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17611577654272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚827405754795508 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚834657904772608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121030656 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47892089 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚834657931164160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚834657904772608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118243079 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚773063027873280 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111178245 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118242396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70350990 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47891411 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12776725224896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111178239 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131550571 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚847457098534912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚847453461469184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 trainer/global_step‚5249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12795556696576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚814632638233504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242755182 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242728810 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131576937 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚755451380391424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12799167370752 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242755182 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242728810 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131576937 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚755451380391424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12799167370752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884793001288166e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚834657931164160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242755182 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17611577654272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚827405754795508 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚834657904772608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121030656 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47892089 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚834657931164160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚834657904772608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118243079 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚773063027873280 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111178245 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118242396 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70350990 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47891411 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12776725224896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111178239 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131550571 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚847457098534912 MCDeviceStatsMonitor.oojôÌn_train_batch_end/allocation.small_pool.current‚26366 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚847453461469184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 trainer/global_step‚5249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12795556696576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚814632638233504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187®Ù{92ô- _wandbruntime‚9506  _runtime‚9506!ëº$\õÖ _step‚431 2loss/SRqqt2/assignment_loss‚2.0478328336720115 /loss/FRt1/detection_loss‚1.4567444742660882 /loss/FRt4/detection_loss‚3.0208405025543703 2loss/SRqqt4/assignment_loss‚1.6518656973575436 /loss/FBt3/detection_loss‚2.6695994436740875 /loss/FRt2/assignment_loss‚2.670638721878246 1loss/SRqqt1/assignment_loss‚1.157273493357934 0loss/FBt4/assignment_loss‚0.0857908604273689 %loss/total_loss‚46.91791095150935 epoch‚4 1loss/FBt3/assignment_loss‚0.39304856982198544 1loss/SRqqt4/detection_loss‚1.9869273512862449 /loss/FBt4/detection_loss‚2.4516687790552774 .loss/FRt2/detection_loss‚1.750628979213662 0loss/FRt4/assignment_loss‚3.3395750368795087 2loss/SRqqt3/assignment_loss‚1.6471135536658352 0loss/SRqqt3/detection_loss‚2.564711936588945 /loss/FBt2/detection_loss‚0.6175894993768238 0loss/FRt1/assignment_loss‚1.8064863942563534 /loss/FRt3/assignment_loss‚5.621091842651367 1loss/FBt2/assignment_loss‚0.11226502833511214 1loss/SRqqt1/detection_loss‚0.9676566103162865 trainer/global_step‚5249 1loss/FBt1/assignment_loss‚0.24066212813158927 .loss/FRt3/detection_loss‚3.655655810707494 % _timestamp‚1.7884793001306467e+09 _runtime‚9506.886063068 1loss/SRqqt2/detection_loss‚2.3835428997322365 /loss/FBt1/detection_loss‚2.6187005043029785 GAaöÛ /loss/FBt4/detection_loss‚2.4516687790552774 .loss/FRt2/detection_loss‚1.750628979213662 0loss/FRt4/assignment_loss‚3.3395750368795087 2loss/SRqqt3/assignment_loss‚1.6471135536658352 0loss/SRqqt3/detection_loss‚2.564711936588945 /loss/FBt2/detection_loss‚0.6175894993768238 0loss/FRt1/assignment_loss‚1.8064863942563534 /loss/FRt3/assignment_loss‚5.621091842651367 1loss/FBt2/assignment_loss‚0.11226502833511214 1loss/SRqqt1/detection_loss‚0.9676566103162865 trainer/global_step‚5249 1loss/FBt1/assignment_loss‚0.24066212813158927 .loss/FRt3/detection_loss‚3.655655810707494 % _timestamp‚1.7884793001306467e+09 _runtime‚9506.886063068 1loss/SRqqt2/detection_loss‚2.3835428997322365 /loss/FBt1/detection_loss‚2.6187005043029785 _step‚431 2loss/SRqqt2/assignment_loss‚2.0478328336720115 /loss/FRt1/detection_loss‚1.4567444742660882 /loss/FRt4/detection_loss‚3.0208405025543703 2loss/SRqqt4/assignment_loss‚1.6518656973575436 /loss/FBt3/detection_loss‚2.6695994436740875 /loss/FRt2/assignment_loss‚2.670638721878246 1loss/SRqqt1/assignment_loss‚1.157273493357934 0loss/FBt4/assignment_loss‚0.0857908604273689 %loss/total_loss‚46.91791095150935 epoch‚4 1loss/FBt3/assignment_loss‚0.39304856982198544 1loss/SRqqt4/detection_loss‚1.9869273512862449¯]‚Œ÷‚(: ÇŽèÔÑûžx gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.4' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.68 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂE‹Å•ø‚(:Š ÇŽèÔìËÃx( proc.memory.percent‚19.9873685836792 cpu‚25.211422010126416 proc.cpu.threads‚17% disk./.usageGB‚1161.2074737548828! disk.nvme0c0n1.in‚ 23976.90625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 288.66015625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.32421875& disk.nvme0n1.out‚230047.1962890625 disk.sda2.out‚0 network.sent‚ 1.4803474e+07 network.recv‚ 2.996435e+06# disk.dm-0.out‚230047.1962890625 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.nvme0n1.in‚ 23976.90625 disk.sda1.in‚0# proc.memory.rssMB‚ 3274.73046875) disk./.usagePercent‚77.92960788165685( disk.nvme0c0n1.out‚230047.1962890625 disk.dm-0.in‚ 23976.90625 disk.sda3.out‚ 288.66015625 disk.sda1.out‚0$ memory_percent‚99.94032382965088& proc.memory.availableMB‚ 9.77734375 disk.dm-1.out‚ 288.66015625 disk.sda2.in‚0ú_iœù‚(:‘ ÖŽèÔúºäx gpu.0.gpu‚60.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.184' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.436800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=Ó˜ú‚(: ÖŽèÔ¬¤ø{# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 network.recv‚ 3.001805e+06 disk.dm-1.out‚ 289.45703125 disk.sda.out‚ 289.45703125 disk.sda3.in‚ 0.00390625" disk.nvme0n1.in‚24036.78515625 cpu‚25.21122990879258( disk./.usagePercent‚77.9584102653041 disk.sda1.in‚0 disk.dm-0.in‚24036.78515625" disk.dm-0.out‚230395.099609375 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.4911102e+07' disk.nvme0c0n1.out‚230395.099609375 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚1.328125% disk.nvme0n1.out‚230395.099609375 disk.sda2.out‚0$ memory_percent‚99.93770122528076% disk./.usageGB‚1161.6366500854492 disk.sda1.out‚0$ disk.nvme0c0n1.in‚24036.78515625 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 289.45703125' proc.memory.availableMB‚ 10.20703125fŨבû‚(:† åŽèÔ§ú‹w gpu.0.gpu‚85.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚215.065' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.013 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œŠ°¬˜ü‚(: åŽèÔÔØÌy' disk.nvme0c0n1.out‚230684.255859375" disk.dm-0.out‚230684.255859375# proc.memory.rssMB‚ 3274.73046875 network.sent‚ 1.4917285e+07% disk./.usageGB‚1161.4207649230957 disk.sda.in‚ 0.00390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚24112.94140625$ memory_percent‚99.93774890899658( proc.memory.percent‚19.9873685836792) disk./.usagePercent‚77.94392203091883 disk.dm-1.out‚ 289.69921875 disk.sda3.out‚ 289.69921875 disk.drbd1260.in‚0 disk.drbd1260.out‚1.328125' proc.memory.availableMB‚ 10.19921875 network.recv‚ 3.006487e+06 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 289.69921875% disk.nvme0n1.out‚230684.255859375 cpu‚25.2109915089668 proc.cpu.threads‚17 disk.sda3.in‚ 0.00390625" disk.nvme0n1.in‚24112.94140625 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚24112.94140625ιýýý‚(:… ôŽèÔ’…ªy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.006' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6012 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>¤ãþ‚(:’ ôŽèÔñïøy cpu‚25.21076536593974 network.recv‚ 3.009942e+06 disk.dm-0.in‚24112.94140625' disk.nvme0n1.out‚230815.16845703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 1.4922367e+07 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 290.30078125 disk.sda3.in‚ 0.00390625$ memory_percent‚99.93939399719238 proc.cpu.threads‚17) disk./.usagePercent‚77.94387006138938 disk.dm-1.out‚ 290.30078125 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 290.30078125 disk.drbd1260.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 9.9296875# proc.memory.rssMB‚ 3274.73046875% disk./.usageGB‚1161.4199905395508$ disk.nvme0c0n1.in‚24112.94140625) disk.nvme0c0n1.out‚230815.16845703125$ disk.dm-0.out‚230815.16845703125 disk.drbd1260.out‚1.328125" disk.nvme0n1.in‚24112.94140625( proc.memory.percent‚19.9873685836792Ú¢?Q”ÿ‚(:‰ ƒèÔáݵx disk.sda2.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3274.73046875$ disk.nvme0c0n1.in‚24112.94140625 disk.dm-0.in‚24112.94140625 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 290.87890625' disk.nvme0n1.out‚231066.68017578125 disk.sda1.out‚0( proc.memory.percent‚19.9873685836792 cpu‚25.21054998381267 disk.sda3.in‚ 0.00390625) disk./.usagePercent‚77.97755962886527$ disk./.usageGB‚1161.921989440918$ disk.dm-0.out‚231066.68017578125 disk.sda.out‚ 290.87890625! memory_percent‚99.93896484375 disk.dm-1.out‚ 290.87890625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.33203125" disk.nvme0n1.in‚24112.94140625 proc.memory.availableMB‚10 proc.cpu.threads‚17 network.sent‚ 1.4927804e+07 network.recv‚ 3.013956e+06) disk.nvme0c0n1.out‚231066.68017578125 disk.sda.in‚ 0.00390625 disk.sda2.in‚0þb†’€‚(:‡ ƒèÔØ£Õx gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚200.328' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.0656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òâþz–‚(:‹ ’èÔ¾²¿x disk.dm-1.out‚ 291.51953125 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.33203125$ disk.dm-0.out‚231575.68896484375 disk.sda3.in‚ 0.00390625" disk.nvme0n1.in‚24112.94140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3274.73046875) disk./.usagePercent‚77.97795157640019 disk.sda.out‚ 291.51953125' disk.nvme0n1.out‚231575.68896484375 proc.memory.availableMB‚10 cpu‚25.210326668955872) disk.nvme0c0n1.out‚231575.68896484375 disk.dm-0.in‚24112.94140625 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 291.51953125! memory_percent‚99.93896484375( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17 network.sent‚ 1.4932824e+07 network.recv‚ 3.017305e+06% disk./.usageGB‚1161.9278297424316$ disk.nvme0c0n1.in‚24112.94140625ZcI«›‚‚(: ’èÔÍ·øy gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y4W2ƒ- _wandbruntime‚9594  _runtime‚9594”ãåòÅ„¿ & lr-AdamW/pg1‚0.0004498152789879717 & lr-AdamW/pg2‚0.0004498152789879717 trainer/global_step‚5299 # _timestamp‚1.78847938725536e+09 _runtime‚9594.010967933 _step‚432ªÂê~Ê…Ä & lr-AdamW/pg1‚0.0004498152789879717 & lr-AdamW/pg2‚0.0004498152789879717 trainer/global_step‚5299 # _timestamp‚1.78847938725536e+09 _runtime‚9594.010967933 _step‚432°å‹Ÿˆ2†- _wandbruntime‚9594  _runtime‚9594¡Ð(‡‹P XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855110326929920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244936721 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12917295372800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132757706 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚821998796477136 trainer/global_step‚5299 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112179021 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842193031557120 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788479387256592e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚762276609452032 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132784046 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855106689877504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚842193005165568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132784046 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119301098 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚834889810063394 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12894647676910 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚780051317453824 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 _step‚433 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12913684711936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132757706 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112179015 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112179015 _runtime‚9594.012284695 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚821998771049168 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119301827 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚729 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚724 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12917295372800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48323392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780051192215552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12891039014226]Ýtü¥$ˆP RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚821998771049168 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119301827 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚729 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚724 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12917295372800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48323392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚780051192215552 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12891039014226 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855110326929920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244936721 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12917295372800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132757706 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚821998796477136 trainer/global_step‚5299 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112179021 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚842193031557120 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788479387256592e+09 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚762276609452032 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132784046 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855106689877504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚842193005165568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132784046 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119301098 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚834889810063394 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12894647676910 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚780051317453824 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 _step‚433 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12913684711936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132757706 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112179015 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112179015 _runtime‚9594.012284695±Ð,r!2‰- _wandbruntime‚9595  _runtime‚9595o¼O'Š–N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132808639 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48331469 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12893445317512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 trainer/global_step‚5299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12916095193600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245008084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70991235 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚762415899841024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112199445 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780193946075648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17777972212736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132782273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119322704 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚780193820850688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842346809116160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112199439 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12919705867776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842346782724608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112199439 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17778046234624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚9595.565692067 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 245008084 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132782273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70991240 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855266514983936 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚835042545607816 _step‚434&<"ò!'‹›N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚835042545607816 _step‚434 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132808639 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48331469 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12893445317512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 trainer/global_step‚5299 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12916095193600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245008084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70991235 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚762415899841024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112199445 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780193946075648 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17777972212736 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132782273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119322704 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚780193820850688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842346809116160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚708 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112199439 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12919705867776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842346782724608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112199439 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17778046234624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚9595.565692067 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 245008084 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132782273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70991240 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855266514983936 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0²LµÙþ2Œ- _wandbruntime‚9595  _runtime‚9595$RUjZÔ 2loss/SRqqt4/assignment_loss‚2.2614969349113005 trainer/global_step‚5299 0loss/FRt1/assignment_loss‚1.4279660681883495 2loss/SRqqt3/assignment_loss‚1.8161741408209007 0loss/FBt1/assignment_loss‚0.3564842131620805 1loss/SRqqt1/detection_loss‚1.0341420384204907 1loss/SRqqt2/assignment_loss‚1.896330577986581 1loss/FBt2/assignment_loss‚0.24806435382502995 0loss/FBt4/assignment_loss‚0.2139887125390074 /loss/FBt1/detection_loss‚2.4518246054649353 /loss/FBt2/detection_loss‚0.6020405793535537 0loss/FRt2/assignment_loss‚2.7667887104882136 /loss/FRt4/assignment_loss‚3.331565553943316 1loss/FBt3/assignment_loss‚0.47100879044209204 /loss/FRt1/detection_loss‚1.7907314635347575 /loss/FRt2/detection_loss‚2.0297124513321454 epoch‚4 1loss/SRqqt1/assignment_loss‚1.095533311439921 /loss/FRt3/detection_loss‚3.3069809436798097 1loss/SRqqt3/detection_loss‚2.3312929612067013 0loss/SRqqt4/detection_loss‚2.583718087138801 $ _timestamp‚1.788479388812797e+09 _runtime‚9595.568948555 0loss/FRt3/assignment_loss‚3.8254672586917877 0loss/SRqqt2/detection_loss‚2.325565074171339 /loss/FBt3/detection_loss‚2.0838856571599056 %loss/total_loss‚46.83041031793938 _step‚435 /loss/FRt4/detection_loss‚3.4254119793574014 .loss/FBt4/detection_loss‚3.154235850680958$O&«_ŽÙ /loss/FBt2/detection_loss‚0.6020405793535537 0loss/FRt2/assignment_loss‚2.7667887104882136 /loss/FRt4/assignment_loss‚3.331565553943316 1loss/FBt3/assignment_loss‚0.47100879044209204 /loss/FRt1/detection_loss‚1.7907314635347575 /loss/FRt2/detection_loss‚2.0297124513321454 epoch‚4 1loss/SRqqt1/assignment_loss‚1.095533311439921 /loss/FRt3/detection_loss‚3.3069809436798097 1loss/SRqqt3/detection_loss‚2.3312929612067013 0loss/SRqqt4/detection_loss‚2.583718087138801 $ _timestamp‚1.788479388812797e+09 _runtime‚9595.568948555 0loss/FRt3/assignment_loss‚3.8254672586917877 0loss/SRqqt2/detection_loss‚2.325565074171339 /loss/FBt3/detection_loss‚2.0838856571599056 %loss/total_loss‚46.83041031793938 _step‚435 /loss/FRt4/detection_loss‚3.4254119793574014 .loss/FBt4/detection_loss‚3.154235850680958 2loss/SRqqt4/assignment_loss‚2.2614969349113005 trainer/global_step‚5299 0loss/FRt1/assignment_loss‚1.4279660681883495 2loss/SRqqt3/assignment_loss‚1.8161741408209007 0loss/FBt1/assignment_loss‚0.3564842131620805 1loss/SRqqt1/detection_loss‚1.0341420384204907 1loss/SRqqt2/assignment_loss‚1.896330577986581 1loss/FBt2/assignment_loss‚0.24806435382502995 0loss/FBt4/assignment_loss‚0.2139887125390074 /loss/FBt1/detection_loss‚2.4518246054649353³ÁzV^‚(:’ ¡èÔì»ðx proc.cpu.threads‚17 network.recv‚ 3.021798e+06) disk.nvme0c0n1.out‚231923.36865234375 disk.dm-0.in‚24246.76171875 disk.sda.in‚ 0.00390625 disk.drbd1260.out‚ 1.3359375" disk.nvme0n1.in‚24246.76171875' disk.nvme0n1.out‚231923.36865234375$ memory_percent‚99.94215965270996 network.sent‚ 1.4938864e+07# disk./.usageGB‚1161.90523147583 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 292.00390625 disk.sda.out‚ 292.00390625 disk.drbd1260.in‚0% proc.memory.availableMB‚ 9.4765625# proc.memory.rssMB‚ 3274.73046875) disk./.usagePercent‚77.97643498776733$ disk.dm-0.out‚231923.36865234375 disk.sda3.out‚ 292.00390625 disk.sda2.out‚0 disk.sda1.out‚0( proc.memory.percent‚19.9873685836792$ disk.nvme0c0n1.in‚24246.76171875 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda1.in‚0 cpu‚25.2101023134430524±ž®‘‚(:† ¡èÔÓŠŸz gpu.0.gpu‚10.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.873' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1746 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö6ב‘‚(:† °èÔøܽw gpu.0.gpu‚40.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0McK©š’‚(: °èÔ§µ¿y$ disk.dm-0.out‚232173.84130859375 disk.sda.out‚ 292.4140625 disk.sda2.out‚0 disk.sda1.in‚0# memory_percent‚99.9392032623291 network.sent‚ 1.5046409e+07$ disk.nvme0c0n1.in‚24246.76171875) disk.nvme0c0n1.out‚232173.84130859375 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 292.4140625' disk.nvme0n1.out‚232173.84130859375 disk.sda2.in‚0% proc.memory.availableMB‚ 9.9609375 disk.dm-1.out‚ 292.4140625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.3359375# proc.memory.rssMB‚ 3274.73046875 cpu‚25.20992896022709 disk.dm-0.in‚24246.76171875" disk.nvme0n1.in‚24246.76171875 disk.sda1.out‚0( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17 network.recv‚ 3.026847e+06) disk./.usagePercent‚77.97800943410293% disk./.usageGB‚1161.9286918640137i•Ê“‚(:… ¿èÔ ÷¦y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚202.259' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚40.4518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìlm(”‚(:’ ¿èÔˆñÍz network.sent‚ 1.5054777e+07 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 292.9140625' disk.nvme0n1.out‚232424.54541015625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚77.98639291282669 disk.dm-1.out‚ 292.9140625 disk.sda3.in‚ 0.00390625$ disk.nvme0c0n1.in‚24246.76171875$ disk.dm-0.out‚232424.54541015625$ memory_percent‚99.93922710418701& proc.memory.availableMB‚ 9.95703125( proc.memory.percent‚19.9873685836792$ disk./.usageGB‚1162.053611755371 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.33984375" disk.nvme0n1.in‚24246.76171875 disk.sda2.out‚0) disk.nvme0c0n1.out‚232424.54541015625 disk.dm-0.in‚24246.76171875 cpu‚25.209753417832438 network.recv‚ 3.032081e+06 disk.sda3.out‚ 292.9140625# proc.memory.rssMB‚ 3274.73046875 proc.cpu.threads‚17ó [jœ•‚(:‘ ÎèԘƸw gpu.0.gpu‚57.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.044' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.20879999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìë'ç –‚(:• ÎèԵĸy% disk./.usageGB‚1162.1556587219238 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0' proc.memory.availableMB‚ 10.19921875# proc.memory.rssMB‚ 3274.73046875 cpu‚25.20982164560313 disk.dm-0.in‚24246.76171875$ disk.dm-0.out‚232772.30712890625 disk.dm-1.out‚ 293.52734375( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17) disk./.usagePercent‚77.99324137037516" disk.nvme0n1.in‚24246.76171875' disk.nvme0n1.out‚232772.30712890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.drbd1260.out‚ 1.33984375$ memory_percent‚99.93774890899658 network.recv‚ 3.03527e+06 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚24246.76171875) disk.nvme0c0n1.out‚232772.30712890625 disk.sda.out‚ 293.52734375 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 293.52734375 network.sent‚ 1.5057948e+07Kþ,C›—‚(: ÝèÔ¢òþx gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚161.65' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.330000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»7›Ÿ‘˜‚(:† ÝèÔš¡z network.sent‚ 1.5063995e+07 network.recv‚ 3.039863e+06( disk.nvme0c0n1.out‚233093.9931640625 disk.dm-0.in‚ 24399.921875& disk.nvme0n1.out‚233093.9931640625 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.percent‚19.9873685836792 disk.sda1.in‚0 disk.dm-1.out‚ 293.8203125 disk.drbd1260.out‚1.34375$ memory_percent‚99.93922710418701 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 24399.921875 disk.sda.in‚ 0.00390625 disk.nvme0n1.in‚ 24399.921875& proc.memory.availableMB‚ 9.95703125# proc.memory.rssMB‚ 3274.73046875 cpu‚25.209904705656147 proc.cpu.threads‚17) disk./.usagePercent‚78.01162783148429$ disk./.usageGB‚1162.429630279541# disk.dm-0.out‚233093.9931640625 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 293.8203125 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 293.8203125 disk.drbd1260.in‚0,uš’™‚(:‡ ìèÔòþ—y gpu.0.gpu‚50.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚196.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.3632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ˆ|㇚‚(:ü ìèÔÃصz disk.sda3.in‚ 0.00390625& disk.nvme0n1.out‚233512.6923828125 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 network.sent‚ 1.5068983e+07 disk.dm-1.out‚294.1875 disk.sda.out‚294.1875# memory_percent‚99.9392032623291% proc.memory.availableMB‚ 9.9609375 cpu‚25.210016530484932 disk.sda1.in‚0 disk.dm-0.in‚ 24399.921875 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 24399.921875 proc.cpu.threads‚17 network.recv‚ 3.043448e+06) disk./.usagePercent‚78.01156075751032" disk.nvme0c0n1.in‚ 24399.921875( disk.nvme0c0n1.out‚233512.6923828125# disk.dm-0.out‚233512.6923828125 disk.sda.in‚ 0.00390625 disk.sda3.out‚294.1875 disk.drbd1260.out‚1.34375% disk./.usageGB‚1162.4286308288574 disk.sda1.out‚0½ªñY‘›‚(:† ûèÔ•Á£x disk.sda2.in‚0 disk.sda3.out‚ 294.56640625 disk.drbd1260.out‚ 1.34765625# memory_percent‚99.9392032623291% proc.memory.availableMB‚ 9.9609375 cpu‚25.21011480193492 disk.sda1.in‚0 disk.dm-1.out‚ 294.56640625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 294.56640625 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3274.73046875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 24399.921875 disk.sda2.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 24399.921875' disk.nvme0c0n1.out‚233989.533203125 network.recv‚ 3.048295e+06) disk./.usagePercent‚78.01155947747264$ disk./.usageGB‚1162.428611755371 disk.dm-0.in‚ 24399.921875" disk.dm-0.out‚233989.533203125 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625( proc.memory.percent‚19.9873685836792 network.sent‚ 1.5075446e+07% disk.nvme0n1.out‚233989.533203125Äå† œ‚(:…³}ldž ûèÔÉÌx gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«qà‰2- _wandbruntime‚9690  _runtime‚9690R(wÇžÁ & lr-AdamW/pg1‚0.0004454505756537185 & lr-AdamW/pg2‚0.0004454505756537185 trainer/global_step‚5349 % _timestamp‚1.7884794833255167e+09 _runtime‚9690.082042322 _step‚436ˆóÜôÌŸÆ _step‚436 & lr-AdamW/pg1‚0.0004454505756537185 & lr-AdamW/pg2‚0.0004454505756537185 trainer/global_step‚5349 % _timestamp‚1.7884794833255167e+09 _runtime‚9690.082042322´ïç…2 - _wandbruntime‚9690  _runtime‚9690$"«&(¡ŒP H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚769241077698560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚842530234966354 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚849881883117568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13037833869824 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113200215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71618680 _step‚437 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚862916106326528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚787181827896832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113200215 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113200221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚849881883117568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13011367769734 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13014976432418 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113200221 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71618685 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884794833289814e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247215969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48764661 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 _runtime‚9690.086127233 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚862919743378944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134015748 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚829515258533936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13034223208960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17940750198272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚842526600875702 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚829515233105968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚787181951037952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120383346 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133989408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13037833869824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48763960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247189623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚769241128901632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862916106326528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249jf'(¢‘P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 _runtime‚9690.086127233 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚862919743378944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134015748 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚829515258533936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13034223208960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17940750198272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚842526600875702 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚829515233105968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚787181951037952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120383346 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133989408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13037833869824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48763960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247189623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚769241128901632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862916106326528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚769241077698560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚842530234966354 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚849881883117568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13037833869824 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113200215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71618680 _step‚437 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚862916106326528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚787181827896832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113200215 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113200221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚849881883117568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13011367769734 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13014976432418 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113200221 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71618685 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884794833289814e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247215969 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48764661 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚829665562347104 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 % _timestamp‚1.7884794854107535e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113220645 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863075931432960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120404907 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚829665587775072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13036633690624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113220639 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247234614 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48772730 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17944079971840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚850035660676608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134013975 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚787324573284352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120404215 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863072294367232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134013975 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134040341 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247260986 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚850035687068160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13017382735912 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚787324448059392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚842679336420124 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247260986 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113220645 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚769380419290624 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5349 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13040244364800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚850035660676608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247234614 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚769380368087552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372–tTþÆ¥›N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚769380368087552 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚850035687068160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚829665562347104 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.ªž¾["all.allocated‚ 33395048448 % _timestamp‚1.7884794854107535e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113220645 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863075931432960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120404907 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚829665587775072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13036633690624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113220639 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247234614 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48772730 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17944079971840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚850035660676608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134013975 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚787324573284352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120404215 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚863072294367232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134013975 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚692 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134040341 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247260986 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚850035687068160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13017382735912 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚787324448059392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚842679336420124 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247260986 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113220645 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚769380419290624 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚5349 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13040244364800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚850035660676608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247234614¶§ï42¦- _wandbruntime‚9692  _runtime‚9692F™±¬V§Ð /loss/FRt4/assignment_loss‚2.679517265628366 2loss/SRqqt3/assignment_loss‚1.3411810917158922 .loss/FRt4/detection_loss‚2.934727370738983 .loss/FBt4/detection_loss‚2.691144287586212 % _timestamp‚1.7884794854131324e+09 2loss/SRqqt4/assignment_loss‚1.7019089334644377 1loss/FBt4/assignment_loss‚0.18199083769468416 .loss/FRt2/detection_loss‚1.945421706540165 1loss/SRqqt3/detection_loss‚2.4241549114386243 .loss/FBt3/detection_loss‚2.669421999454498 1loss/FBt3/assignment_loss‚0.23722072369477246 trainer/global_step‚5349 /loss/FBt2/detection_loss‚0.8604018706190464 %loss/total_loss‚45.56455589356663 2loss/SRqqt1/assignment_loss‚0.9828087903344287 _step‚439 /loss/FRt2/assignment_loss‚2.220748040411207 0loss/FRt3/assignment_loss‚5.2068671703338625 1loss/SRqqt2/assignment_loss‚2.487076065119575 0loss/SRqqt2/detection_loss‚2.770395692657022 .loss/FBt1/detection_loss‚2.354210620125135 epoch‚4 _runtime‚ 9692.16850479 0loss/FRt1/assignment_loss‚1.5079904281430774 1loss/FBt2/assignment_loss‚0.22786236790204986 1loss/SRqqt1/detection_loss‚0.8496063278844723 0loss/FBt1/assignment_loss‚0.4325122081728144 /loss/FRt1/detection_loss‚1.4259193449866467 .loss/FRt3/detection_loss‚3.128560736179352 0loss/SRqqt4/detection_loss‚2.302907102741301^KíÔ[¨Õ epoch‚4 _runtime‚ 9692.16850479 0loss/FRt1/assignment_loss‚1.5079904281430774 1loss/FBt2/assignment_loss‚0.22786236790204986 1loss/SRqqt1/detection_loss‚0.8496063278844723 0loss/FBt1/assignment_loss‚0.4325122081728144 /loss/FRt1/detection_loss‚1.4259193449866467 .loss/FRt3/detection_loss‚3.128560736179352 0loss/SRqqt4/detection_loss‚2.302907102741301 /loss/FRt4/assignment_loss‚2.679517265628366 2loss/SRqqt3/assignment_loss‚1.3411810917158922 .loss/FRt4/detection_loss‚2.934727370738983 .loss/FBt4/detection_loss‚2.691144287586212 % _timestamp‚1.7884794854131324e+09 2loss/SRqqt4/assignment_loss‚1.7019089334644377 1loss/FBt4/assignment_loss‚0.18199083769468416 .loss/FRt2/detection_loss‚1.945421706540165 1loss/SRqqt3/detection_loss‚2.4241549114386243 .loss/FBt3/detection_loss‚2.669421999454498 1loss/FBt3/assignment_loss‚0.23722072369477246 trainer/global_step‚5349 /loss/FBt2/detection_loss‚0.8604018706190464 %loss/total_loss‚45.56455589356663 2loss/SRqqt1/assignment_loss‚0.9828087903344287 _step‚439 /loss/FRt2/assignment_loss‚2.220748040411207 0loss/FRt3/assignment_loss‚5.2068671703338625 1loss/SRqqt2/assignment_loss‚2.487076065119575 0loss/SRqqt2/detection_loss‚2.770395692657022 .loss/FBt1/detection_loss‚2.354210620125135·e ž–©‚(:‹ ŠèÔ¥¨þx disk.sda1.out‚0 disk.dm-0.in‚ 24399.921875# proc.memory.rssMB‚ 3274.73046875 cpu‚25.210197552845845 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.34765625 disk.nvme0n1.in‚ 24399.921875 network.sent‚ 1.5080295e+07 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚234263.23388671875 disk.sda3.out‚ 294.9765625' disk.nvme0n1.out‚234263.23388671875$ memory_percent‚99.93958473205566% proc.memory.availableMB‚ 9.8984375( proc.memory.percent‚19.9873685836792% disk./.usageGB‚1162.9306297302246 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 294.9765625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 294.9765625 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 24399.921875$ disk.dm-0.out‚234263.23388671875 network.recv‚ 3.051523e+06( disk./.usagePercent‚78.0452503249862Õ;Rœª‚(:‘ ŠèÔÍ‘¤y gpu.0.gpu‚24.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚264.184' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.836800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚Zˆd«‚(:’ ™èÔɹz gpu.0.gpu‚43.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚138.585' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.717000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ–o’¬‚(:‡ ™èÔ˜©¨{) disk.nvme0c0n1.out‚234538.32373046875 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3274.73046875 cpu‚25.21022037236983 network.sent‚ 1.5189258e+07) disk./.usagePercent‚78.04511489700059 disk.sda2.out‚0 disk.sda.out‚ 295.34375 disk.sda3.in‚ 0.00390625( proc.memory.percent‚19.9873685836792$ disk./.usageGB‚1162.928611755371 disk.sda1.out‚0 disk.dm-1.out‚ 295.34375 disk.sda3.out‚ 295.34375 disk.drbd1260.out‚ 1.3515625 proc.cpu.threads‚17 network.recv‚ 3.058033e+06' disk.nvme0n1.out‚234538.32373046875 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 24518.9765625$ disk.dm-0.out‚234538.32373046875$ memory_percent‚99.93970394134521& proc.memory.availableMB‚ 9.87890625! disk.nvme0n1.in‚ 24518.9765625# disk.nvme0c0n1.in‚ 24518.9765625CL‘­‚(:† ¨èÔš„ùy gpu.0.gpu‚12.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚129.639' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9278 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÎR“®‚(:ˆ ¨èÔ»¹õz disk.dm-0.in‚ 24518.9765625 network.sent‚ 1.5194278e+07) disk./.usagePercent‚78.04511489700059 disk.dm-1.in‚ 0.00390625! disk.nvme0n1.in‚ 24518.9765625 disk.sda2.in‚0! disk.dm-0.out‚235053.63671875$ memory_percent‚99.94113445281982# proc.memory.rssMB‚ 3274.73046875$ disk./.usageGB‚1162.928611755371 disk.dm-1.out‚ 295.72265625# disk.nvme0c0n1.in‚ 24518.9765625& proc.memory.availableMB‚ 9.64453125( proc.memory.percent‚19.9873685836792 cpu‚25.210334110535218 proc.cpu.threads‚17 network.recv‚ 3.061263e+06 disk.sda3.in‚ 0.00390625 disk.drbd1260.out‚ 1.3515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.00390625 disk.sda.out‚ 295.72265625 disk.sda3.out‚ 295.72265625 disk.drbd1260.in‚0$ disk.nvme0n1.out‚235053.63671875 disk.sda2.out‚0& disk.nvme0c0n1.out‚235053.63671875ñ^×Ò˜¯‚(: ·èÔÆï§x$ proc.memory.availableMB‚9.890625 network.sent‚ 1.5200743e+07! disk.nvme0n1.in‚ 24518.9765625 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 24518.9765625# proc.memory.rssMB‚ 3274.73046875( proc.memory.percent‚19.9873685836792 cpu‚25.21038317145807 disk.sda2.in‚0# disk.dm-0.out‚235399.6806640625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 296.03515625 disk.sda.out‚ 296.03515625 disk.drbd1260.out‚ 1.35546875$ memory_percent‚99.93963241577148% disk./.usageGB‚1162.9286346435547 disk.sda1.in‚0( disk.nvme0c0n1.out‚235399.6806640625 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.recv‚ 3.066021e+06) disk./.usagePercent‚78.04511643304579& disk.nvme0n1.out‚235399.6806640625 disk.sda1.out‚0 disk.dm-0.in‚ 24518.9765625 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 296.03515625ø˜·>‘°‚(:† ·èÔçÏ¿x gpu.0.gpu‚29.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚112.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yR0Ò•±‚(:Š ÆèÔˆŠëx disk.sda.out‚ 296.6015625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 24518.9765625 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 24518.9765625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 296.6015625 disk.drbd1260.out‚ 1.35546875 disk.sda1.out‚0$ proc.memory.availableMB‚9.640625( proc.memory.percent‚19.9873685836792 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 296.6015625# proc.memory.rssMB‚ 3274.73046875 cpu‚25.21046591241258) disk./.usagePercent‚78.04354301074035( disk.nvme0c0n1.out‚235842.3642578125 disk.dm-0.in‚ 24518.9765625 disk.sda3.in‚ 0.00390625& disk.nvme0n1.out‚235842.3642578125 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚99.94115829467773 network.sent‚ 1.5205562e+07 network.recv‚ 3.069335e+06% disk./.usageGB‚1162.9051895141602# disk.dm-0.out‚235842.3642578125•ï‡ Œ²‚(: ÆèÔ—ëÀy gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.4' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.68 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P½s2³- _wandbruntime‚9777  _runtime‚9777ðÈûÈÉ´Ã _runtime‚9777.733810475 _step‚440 ' lr-AdamW/pg1‚0.00044106096358739126 ' lr-AdamW/pg2‚0.00044106096358739126 trainer/global_step‚5399 % _timestamp‚1.7884795709781985e+09ë®QÎµÈ ' lr-AdamW/pg1‚0.00044106096358739126 ' lr-AdamW/pg2‚0.00044106096358739126 trainer/global_step‚5399 % _timestamp‚1.7884795709781985e+09 _runtime‚9777.733810475 _step‚440¸ƒÉ¬2¶- _wandbruntime‚9777  _runtime‚9777oÌÅô(·ŒP ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135221110 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121464097 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18106841888768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72258930 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚794312367098880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚870725522775552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249468871 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135247450 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121463383 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚837031720590736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13131696525242 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚776205648351232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114221421 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚857570787461120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72258935 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18106769950720 _runtime‚9777.735247826 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13158372366848 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚850163391688010 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114221421 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49204453 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚870725522775552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚837031695162768 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135247450 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13154761705984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13154761705984 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114221415 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249442525 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135221110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚850167025778662 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13135305187926 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 % _timestamp‚1.7884795709795153e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚857570787461120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49205162 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5399 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚870729159827968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚870725522775552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚837031695162768 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135247450 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13154761705984 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13154761705984 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114221415 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249442525 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135221110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚850167025778662 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ØË%½13135305187926 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 % _timestamp‚1.7884795709795153e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚857570787461120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49205162 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5399 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚870729159827968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18106841888768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72258930 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚794312367098880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚870725522775552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249468871 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135247450 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121463383 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚837031720590736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13131696525242 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚776205648351232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114221421 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚857570787461120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72258935 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18106769950720 _runtime‚9777.735247826 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187¹Ö»û2¹- _wandbruntime‚9778  _runtime‚9778Nƒ |'º–N P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114241839 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18110086793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249513888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114241845 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121485721 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121485031 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚850316127232432 _step‚442 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚857724538628608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135245677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49213296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249513888 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚837182024403904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚870885347881984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135272043 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚794455097458688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚794454974330880 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚857724538628608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 % _timestamp‚1.7884795720388806e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚776344887537152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18110158718464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114241839 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚850319761323292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249487516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13160782861824 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114241845 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49213981 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13157172187648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚857724565020160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249487516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚837182049831872`UíF!'»›N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135245677 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13160782861824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚870885347881984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114241839 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18110086793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249513888 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114241845 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121485721 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121485031 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚685 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚850316127232432 _step‚442 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚857724538628608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135245677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49213296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249513888 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚837182024403904 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚870885347881984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135272043 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚794455097458688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚794454974330880 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚857724538628608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 % _timestamp‚1.7884795720388806e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚776344887537152 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18110158718464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114241839 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚850319761323292 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249487516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13160782861824 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114241845 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49213981 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13157172187648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚857724565020160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249487516 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚837182049831872 PÀ‚(:„ ÕèÔ¥½íz disk.drbd1260.in‚0$ disk.nvme0n1.out‚236150.99609375$ memory_percent‚99.94158744812012 network.sent‚ 1.5211711e+07 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 297.06640625 disk.drbd1260.out‚1.359375 disk.sda2.in‚0! proc.memory.rssMB‚ 3274.796875 cpu‚25.210388631609323 proc.cpu.threads‚17) disk./.usagePercent‚77.96744502719496 disk.sda1.out‚0% proc.memory.availableMB‚ 9.5703125* proc.memory.percent‚19.987773895263672 network.recv‚ 3.073849e+06" disk.nvme0c0n1.in‚ 24627.171875& disk.nvme0c0n1.out‚236150.99609375! disk.dm-0.out‚236150.99609375 disk.sda.in‚ 0.00390625 disk.nvme0n1.in‚ 24627.171875 disk.sda2.out‚0 disk.sda1.in‚0% disk./.usageGB‚1161.7712745666504 disk.dm-0.in‚ 24627.171875 disk.dm-1.out‚ 297.06640625 disk.sda.out‚ 297.06640625 disk.sda3.in‚ 0.00390625¸¤†’Á‚(:‡ äèÔ—߈w gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l0û…‚(:ú äèÔ‘Èéy& proc.memory.availableMB‚ 9.28515625 network.sent‚ 1.5319247e+07 network.recv‚ 3.078839e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 297.5078125 disk.drbd1260.out‚1.359375 cpu‚25.2102267440975% disk./.usageGB‚1161.8664512634277! proc.memory.rssMB‚ 3274.796875* proc.memory.percent‚19.987773895263672 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 236441.015625 disk.dm-0.in‚ 24627.171875 disk.dm-0.out‚ 236441.015625) disk./.usagePercent‚77.97383241517426" disk.nvme0n1.out‚ 236441.015625 disk.sda2.in‚0 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.nvme0n1.in‚ 24627.171875" disk.nvme0c0n1.in‚ 24627.171875 disk.dm-1.out‚ 297.5078125 disk.sda.out‚ 297.5078125 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0$ memory_percent‚99.94332790374756Ç}A&’Â(:‡ óèÔ©ÍÖx gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚257.838' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“3yZÆÄ‚(:‹ óèÔéû£y) disk./.usagePercent‚77.97383215916672 disk.sda.out‚ 297.87890625 disk.sda3.out‚ 297.87890625 disk.dm-0.in‚ 24627.171875 disk.dm-1.out‚ 297.87890625 nëê Ðetwork.sent‚ 1.532576e+07% disk./.usageGB‚1161.8664474487305 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.36328125 disk.nvme0n1.in‚ 24627.171875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 3.08351e+06 disk.sda3.in‚ 0.00390625( disk.nvme0c0n1.out‚236753.9482421875# disk.dm-0.out‚236753.9482421875$ memory_percent‚99.94041919708252& proc.memory.availableMB‚ 9.76171875! proc.memory.rssMB‚ 3274.796875 cpu‚25.210037483270206& disk.nvme0n1.out‚236753.9482421875 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 24627.171875 disk.dm-1.in‚ 0.00390625* proc.memory.percent‚19.987773895263672 proc.cpu.threads‚17j”ƒß’Å‚(:‡ ‚‘èÔÄüÁx gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚275.721' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ’ 4—Æ‚(:Œ ‚‘èÔàÕ y disk.sda.in‚ 0.00390625 disk.sda.out‚ 298.29296875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 298.29296875" disk.nvme0c0n1.in‚ 24627.171875 disk.dm-0.in‚ 24627.171875# memory_percent‚99.9406099319458 cpu‚25.209871156192005 proc.cpu.threads‚17 network.recv‚ 3.086835e+06% disk./.usageGB‚1161.8664474487305 disk.nvme0n1.in‚ 24627.171875& disk.nvme0n1.out‚237209.8583984375( disk.nvme0c0n1.out‚237209.8583984375# disk.dm-0.out‚237209.8583984375* proc.memory.percent‚19.987773895263672 network.sent‚ 1.5330499e+07 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 298.29296875& proc.memory.availableMB‚ 9.73046875! proc.memory.rssMB‚ 3274.796875) disk./.usagePercent‚77.97383215916672 disk.drbd1260.out‚ 1.36328125 disk.sda2.in‚0 disk.sda2.out‚0’ÉÎY”Ç‚(:‰ ‘‘èÔ¹¹Çw% disk./.usageGB‚1161.8655700683594 disk.drbd1260.out‚ 1.3671875 disk.sda1.in‚0 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 298.8203125& proc.memory.availableMB‚ 9.73828125! proc.memory.rssMB‚ 3274.796875* proc.memory.percent‚19.987773895263672$ disk.nvme0n1.out‚237402.54296875 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚24651.49609375! disk.dm-0.out‚237402.54296875 disk.dm-1.out‚ 298.8203125 proc.cpu.threads‚17 network.recv‚ 3.091578e+06) disk./.usagePercent‚77.97377327743385 disk.drbd1260.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚24651.49609375& disk.nvme0c0n1.out‚237402.54296875 disk.sda.in‚ 0.00390625 disk.sda.out‚ 298.8203125 network.sent‚ 1.5336689e+07" disk.nvme0n1.in‚24651.49609375 disk.dm-1.in‚ 0.00390625$ memory_percent‚99.94056224822998 cpu‚25.209940738040356çÛo>È‚(:’ ‘‘èÔņæx gpu.0.gpu‚58.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚225.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.101200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*Êó€2É- _wandbruntime‚9841  _runtime‚9841Ñ_Ì ÈÊ trainer/global_step‚5449 $ _timestamp‚1.788479635033042e+09 _runtime‚9841.790352504 _step‚444 ' lr-AdamW/pg1‚0.00043664735757756567 ' lr-AdamW/pg2‚0.00043664735757756567ø7ÍËÇ _step‚444 ' lr-AdamW/pg1‚0.00043664735757756567 ' lr-AdamW/pg2‚0.00043664735757756567 trainer/global_step‚5449 $ _timestamp‚1.788479635033042e+09 _runtime‚9841.790352504¼¶SX_2Ì- _wandbruntime‚9841  _runtime‚9841ßÝÈÆ(ÍŒP TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136479152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49645367 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚878538576276992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚844548182647536 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251695427 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚865259665413120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚783170116597760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18273150085632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13275300203008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚878534939224576 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122543846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136452812 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚857800182500318 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884796350368302e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115242621 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136452812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚878538576276992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _step‚445 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122544552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115242615 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 trainer/global_step‚5449 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚878534939224576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13255633943434 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72899180 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115242621 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚801443192648192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18273076050432 _runtime‚9841.795280125 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13278910863872ئ€¸(ΑP trainer/global_step‚5449 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚878534939224576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13255633943434 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72899180 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115242621 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚801443192648192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18273076050432 _runtime‚9841.795280125 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13278910863872 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136479152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49645367 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚878538576276992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚844548182647536 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251695427 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚865259665413120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚783170116597760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18273150085632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13275300203008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚878534939224576 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122543846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136452812 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚857800182500318 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884796350368302e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115242621 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136452812 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚878538576276992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 _step‚445 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122544552 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115242615 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187½R!N2Ï- _wandbruntime‚9844  _runtime‚9844CQD' ЕN OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115263039 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚857952918044740 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115263045 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49653981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115263045 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚783309406986752 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚844698486460704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115263039 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚865413442972160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49653310 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚878691127265280 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13277710684672 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 trainer/global_step‚5449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13277710684672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13281321358848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72911990 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚446 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚865413442972160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251766790 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚801585809131008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚857956552135600 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136503745 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 _runtime‚ 9844.55265869 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122565295 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136477379 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13258040246928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251766790 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚878694764331008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚878694764331008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136503745 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122565971 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18276402144256 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚865413416580608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_±­ÍÐôend/inactive_split.small_pool.current‚671 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136477379 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚783309458189824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13254431584036 % _timestamp‚1.7884796377909372e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13254431584036 % _timestamp‚1.7884796377909372e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚446 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚865413442972160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251766790 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚801585809131008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚857956552135600 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136503745 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 _runtime‚ 9844.55265869 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122565295 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136477379 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13258040246928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251766790 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚878694764331008¾­(Ò¨2Ò- _wandbruntime‚9844  _runtime‚9844B¹ q[ÓÕ trainer/global_step‚5449 % _timestamp‚1.7884796377996285e+09 0loss/FRt1/assignment_loss‚2.0312377897001084 0loss/FRt3/assignment_loss‚3.5625754594802856 /loss/FRt2/detection_loss‚1.9535126756303585 .loss/FBt3/detection_loss‚1.757754764954249 2loss/SRqqt3/assignment_loss‚1.6309369582505453 1loss/SRqqt2/detection_loss‚2.9950363721166338 /loss/FBt2/detection_loss‚0.7927080867035935 _step‚447 2loss/SRqqt2/assignment_loss‚2.2115150626216615 1loss/FBt1/assignment_loss‚0.09006627338668396 1loss/SRqqt4/detection_loss‚2.2785014477041035 /loss/FBt4/detection_loss‚2.6642626176277795 _runtime‚ 9844.55562834 2loss/SRqqt4/assignment_loss‚1.9907218143343925 1loss/FBt3/assignment_loss‚0.42492435806828627 1loss/FBt4/assignment_loss‚0.09014826271595666 1loss/SRqqt3/detection_loss‚2.7816609144210815 /loss/FRt2/assignment_loss‚2.354201322564712 1loss/FBt2/assignment_loss‚0.14174229406644764 1loss/SRqqt1/detection_loss‚1.0751606814761077 /loss/FBt1/detection_loss‚2.1887127028571234 0loss/FRt4/assignment_loss‚3.6413010478019716 epoch‚4 1loss/SRqqt1/assignment_loss‚1.201543832016466 /loss/FRt1/detection_loss‚1.5747314389914062 -loss/FRt3/detection_loss‚3.12433506300052 .loss/FRt4/detection_loss‚2.851202275071825 $loss/total_loss‚45.4084935155623Œ9T`ÔÚ _step‚447 2loss/SRqqt2/assignment_loss‚2.2115150626216615 1loss/FBt1/assignment_loss‚0.09006627338668396 1loss/SRqqt4/detection_loss‚2.2785014477041035 /loss/FBt4/detection_loss‚2.6642626176277795 _runtime‚ 9844.55562834 2loss/SRqqt4/assignment_loss‚1.9907218143343925 1loss/FBt3/assignment_loss‚0.42492435806828627 1loss/FBt4/assignment_loss‚0.09014826271595666 1loss/SRqqt3/detection_loss‚2.7816609144210815 /loss/FRt2/assignment_loss‚2.354201322564712 1loss/FBt2/assignment_loss‚0.14174229406644764 1loss/SRqqt1/detection_loss‚1.0751606814761077 /loss/FBt1/detection_loss‚2.1887127028571234 0loss/FRt4/assignment_loss‚3.6413010478019716 epoch‚4 1loss/SRqqt1/assignment_loss‚1.201543832016466 /loss/FRt1/detection_loss‚1.5747314389914062 -loss/FRt3/detection_loss‚3.12433506300052 .loss/FRt4/detection_loss‚2.851202275071825 $loss/total_loss‚45.4084935155623 trainer/global_step‚5449 % _timestamp‚1.7884796377996285e+09 0loss/FRt1/assignment_loss‚2.0312377897001084 0loss/FRt3/assignment_loss‚3.5625754594802856 /loss/FRt2/detection_loss‚1.9535126756303585 .loss/FBt3/detection_loss‚1.757754764954249 2loss/SRqqt3/assignment_loss‚1.6309369582505453 1loss/SRqqt2/detection_loss‚2.9950363721166338 /loss/FBt2/detection_loss‚0.7927080867035935¿AîÕ‚(:„  ‘èÔ²÷±x gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚269.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.916 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"·pŒÖ‚(:  ‘èÔõðÄx disk.sda2.out‚0 network.sent‚ 1.5341754e+07 network.recv‚ 3.094911e+06 disk.sda.in‚ 0.00390625$ disk.nvme0n1.out‚237860.61328125 disk.sda2.in‚0* proc.memory.percent‚19.987773895263672 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 24778.0703125& disk.nvme0c0n1.out‚237860.61328125 disk.sda3.out‚ 299.359375 disk.sda1.in‚0 disk.sda1.out‚0$ proc.memory.availableMB‚9.734375) disk./.usagePercent‚77.96707202421757% disk./.usageGB‚1161.7657165527344! disk.dm-0.out‚237860.61328125 disk.dm-1.in‚ 0.00390625 disk.drbd1260.out‚ 1.3671875$ memory_percent‚99.94058609008789! proc.memory.rssMB‚ 3274.796875 cpu‚25.209870386548037 disk.dm-0.in‚ 24778.0703125 disk.dm-1.out‚ 299.359375 disk.sda.out‚ 299.359375 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 24778.0703125ÓRׂ(:‡ ¯‘èÔ©Š¡x gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚282.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û¨›Ø‚(: ¯‘èÔ‹žÃx* proc.memory.percent‚19.987773895263672 cpu‚25.209679915021116$ disk.dm-0.out‚238036.81494140625 disk.drbd1260.out‚ 1.37109375! disk.nvme0n1.in‚ 24778.0703125 disk.sda2.out‚0) disk.nvme0c0n1.out‚238036.81494140625% disk./.usageGB‚1161.9234924316406 disk.dm-0.in‚ 24778.0703125 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda2.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 3274.796875 proc.cpu.threads‚17 disk.sda.out‚ 300.0390625 disk.sda3.out‚ 300.0390625 disk.drbd1260.in‚0' disk.nvme0n1.out‚238036.81494140625# disk.nvme0c0n1.in‚ 24778.0703125$ memory_percent‚99.94194507598877& proc.memory.availableMB‚ 9.51171875 disk.sda3.in‚ 0.00390625 network.sent‚ 1.5450994e+07 network.recv‚ 3.101893e+06) disk./.usagePercent‚77.97766049583376 disk.dm-1.out‚ 300.0390625 disk.sda1.out‚0¡d&íÙ‚(:… ¾‘èÔ›ëªx# disk.nvme0c0n1.in‚ 24778.0703125# memory_percent‚99.9420166015625 network.sent‚ 1.5455844e+07 network.recv‚ 3.105194e+06) disk./.usagePercent‚77.97763822317827 disk.sda.in‚ 0.00390625 disk.drbd1260.out‚ 1.37109375 disk.sda2.in‚0 disk.dm-0.in‚ 24778.0703125$ disk.dm-0.out‚238502.32666015625 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 24778.0703125 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 300.640625 proc.memory.availableMB‚9.5! proc.memory.rssMB‚ 3274.796875* proc.memory.percent‚19.987773895263672 disk.sda.out‚ 300.640625) disk.nvme0c0n1.out‚238502.32666015625 cpu‚25.209741544237094 proc.cpu.threads‚17% disk./.usageGB‚1161.9231605529785 disk.sda3.out‚ 300.640625' disk.nvme0n1.out‚238502.32666015625 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0Û#OÚ‚(:… ¾‘èԹߙy gpu.0.gpu‚6.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é`t›Û‚(: Í‘èÔªé°y gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.184' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.436799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝVK•Ü‚(:Š Í‘èÔÞ¾{ disk.drbd1260.out‚1.375' disk.nvme0n1.out‚238695.17822265625# proc.memory.rssMB‚ 3274.92578125 network.sent‚ 1.5463865e+07$ disk.dm-0.out‚238695.17822265625 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 300.91015625 disk.sda2.out‚0 disk.sda1.out‚0! proc.memory.availableMB‚9.625 cpu‚25.20973549003127% disk./.usageGB‚1161.9231414794922) disk.nvme0c0n1.out‚238695.17822265625 disk.dm-0.in‚ 24778.0703125 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 24778.0703125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.00390625 disk.sda1.in‚0$ memory_percent‚99.94125366210938* proc.memory.percent‚19.988560676574707 disk.sda.out‚ 300.91015625 network.recv‚ 3.109934e+06) disk./.usagePercent‚77.97763694314061# disk.nvme0c0n1.in‚ 24778.0703125 disk.dm-1.out‚ 300.91015625Àá°’Ý‚(:‡ Ü‘èÔøÆèx gpu.0.gpu‚93.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚184.721' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z#”Þ‚(:‰ Ü‘èÔ—œy network.recv‚ 3.11247e+06) disk./.usagePercent‚77.97795413647552 disk.sda.in‚ 0.00390625" disk.nvme0n1.in‚24942.33203125' disk.nvme0n1.out‚239084.82666015625 proc.memory.rssMB‚ 3275.1875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 301.4609375 disk.drbd1260.in‚0# memory_percent‚99.9434232711792& proc.memory.availableMB‚ 9.26953125 proc.cpu.threads‚17 disk.dm-0.in‚24942.33203125$ disk.dm-0.out‚239084.82666015625 disk.dm-1.out‚ 301.4609375 network.sent‚ 1.546587e+07% disk./.usageGB‚1161.9278678894043 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚24942.33203125) disk.nvme0c0n1.out‚239084.82666015625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 301.4609375 disk.drbd1260.out‚1.375* proc.memory.percent‚19.990158081054688 cpu‚25.209757590951067¶à2ß- _wandbruntime‚9916  _runtime‚9916e“ÛÇàÁ trainer/global_step‚5499 $ _timestamp‚1.788479709548962e+09 _runtime‚ 9916.30636501 _step‚448 ' lr-AdamW/pg1‚0.00043221067741311853 ' lr-AdamW/pg2‚0.00043221067741311853¾oX·ÌáÆ _runtime‚ 9916.30636501 _step‚448 ' lr-AdamW/pg1‚0.00043221067741311853 ' lr-AdamW/pg2‚0.00043221067741311853 trainer/global_step‚5499 $ _timestamp‚1.788479709548962e+09À\ 2â- _wandbruntime‚9916  _runtime‚9916ðËBÛ(ãŒP VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚886347992726016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚9916.307101469 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13395838700032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18439358040576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123626882 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116263821 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚737 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116263815 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚790134636047360 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116263815 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 % _timestamp‚1.7884797095503972e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚886344355673600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚872948516973568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18439284005376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13372354036258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚808573920052736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13375962698942 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚5499 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚872948543365120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123626145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137710854 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253948329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13395838700032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚852064619276368 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚865440607403278 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚886344355673600 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137710854 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137684514 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚852064644704336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13399449360896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253974675 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13399449360896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚872948543365120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816à2µø¥ä‘P XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚865440607403278 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚886344355673600 [LDeviceStatsMonitor.onWZßr%_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137710854 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137684514 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚852064644704336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13399449360896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253974675 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13399449360896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚872948543365120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚886347992726016 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 _runtime‚9916.307101469 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13395838700032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18439358040576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123626882 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116263821 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚737 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116263815 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚790134636047360 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116263815 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 % _timestamp‚1.7884797095503972e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚886344355673600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚872948516973568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18439284005376 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13372354036258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚808573920052736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13375962698942 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚5499 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚872948543365120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123626145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137710854 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253948329 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13395838700032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚852064619276368ÁLJ7ü2å- _wandbruntime‚9918  _runtime‚9918:$ªµ'æ–N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 trainer/global_step‚5499 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123648412 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18442609750016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137709081 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚852214948517504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13401859855872 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18442683771904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚450 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137735447 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13401859855872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873102294532608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137709081 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253993320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123647693 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚886504180780032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73552235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚808716536186368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚865589708857048 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73552240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚852214973945472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚886500543714304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13398249181696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚719 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254019692 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960³ðxG!'ç›N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 trainer/global_step‚5499 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123648412 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18442609750016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137709081 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚852214948517504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13401859855872 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18442683771904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚450 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137735447 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13401859855872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873102294532608 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137709081 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253993320 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123647693 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚886504180780032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73552235 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚808716536186368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚865589708857048 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73552240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚852214973945472 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚886500543714304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13398249181696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚719 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254019692 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960ÂEs{‚2è- _wandbruntime‚9918  _runtime‚9918Y`b ]é× /loss/FRt1/detection_loss‚1.7811778220289853 -loss/FBt4/detection_loss‚2.12184001434417 % _timestamp‚1.7884797113918076e+09 2loss/SRqqt4/assignment_loss‚1.6500289804302155 /loss/FBt1/detection_loss‚2.0587281100451946 _runtime‚9918.147500543 _step‚451 0loss/FRt1/assignment_loss‚1.8689571152130762 1loss/FBt2/assignment_loss‚0.24274010650504618 1loss/SRqqt1/detection_loss‚1.0071972328253458 1loss/SRqqt2/detection_loss‚2.9504215766986213 /loss/FBt2/detection_loss‚0.7963053438497681 0loss/FRt4/assignment_loss‚3.4034234038714706 2loss/SRqqt2/assignment_loss‚2.3002606431643167 /loss/FRt2/detection_loss‚1.9063551235944032 /loss/FRt4/detection_loss‚3.1082971897618523 /loss/FBt3/detection_loss‚2.4577118579079125 1loss/FBt3/assignment_loss‚0.44959901665082297 /loss/FRt3/detection_loss‚3.6295803040266037 1loss/SRqqt4/detection_loss‚1.9594372071325779 epoch‚4 /loss/FRt2/assignment_loss‚2.709982480605443 2loss/SRqqt1/assignment_loss‚1.3195881116486414 2loss/SRqqt3/assignment_loss‚1.4881981205323647 trainer/global_step‚5499 0loss/FBt4/assignment_loss‚0.2927616900131051 1loss/SRqqt3/detection_loss‚2.5461825841459733 %loss/total_loss‚45.75765544335602 1loss/FBt1/assignment_loss‚0.02477478646885345 .loss/FRt3/assignment_loss‚3.68410662189126ãyÔÕbêÜ 0loss/FRt1/assignment_loss‚1.8689571152130762 1loss/FBt2/assignment_loss‚0.24274010650504618 1loss/SRqqt1/detection_loss‚1.0071972328253458 1loss/SRqqt2/detection_loss‚2.9504215766986213 /loss/FBt2/detection_loss‚0.7963053438497681 0loss/FRt4/assignment_loss‚3.4034234038714706 2loss/SRqqt2/assignment_loss‚2.3002606431643167 /loss/FRt2/detection_loss‚1.9063551235944032 /loss/FRt4/detection_loss‚3.1082971897618523 /loss/FBt3/detection_loss‚2.4577118579079125 1loss/FBt3/assignment_loss‚0.44959901665082297 /loss/FRt3/detection_loss‚3.6295803040266037 1loss/SRqqt4/detection_loss‚1.9594372071325779 epoch‚4 /loss/FRt2/assignment_loss‚2.709982480605443 2loss/SRqqt1/assignment_loss‚1.3195881116486414 2loss/SRqqt3/assignment_loss‚1.4881981205323647 trainer/global_step‚5499 0loss/FBt4/assignment_loss‚0.2927616900131051 1loss/SRqqt3/detection_loss‚2.5461825841459733 %loss/total_loss‚45.75765544335602 1loss/FBt1/assignment_loss‚0.02477478646885345 .loss/FRt3/assignment_loss‚3.68410662189126 /loss/FRt1/detection_loss‚1.7811778220289853 -loss/FBt4/detection_loss‚2.12184001434417 % _timestamp‚1.7884797113918076e+09 2loss/SRqqt4/assignment_loss‚1.6500289804302155 /loss/FBt1/detection_loss‚2.0587281100451946 _runtime‚9918.147500543 _step‚451Ã<1^öë‚(:‘ ë‘èÔÊ°Ïx network.recv‚ 3.117319e+06) disk./.usagePercent‚77.97795976864128% disk./.usageGB‚1161.9279518127441 disk.dm-1.in‚ 0.00390625 disk.drbd1260.out‚ 1.37890625 disk.sda2.in‚0$ disk.nvme0c0n1.in‚24ΙòN¦942.33203125( disk.nvme0c0n1.out‚239372.6044921875 disk.dm-1.out‚ 301.87109375 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0" disk.nvme0n1.in‚24942.33203125 disk.dm-0.in‚24942.33203125& proc.memory.availableMB‚ 8.96484375" proc.memory.rssMB‚ 3275.4609375 cpu‚25.20982106986302 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚239372.6044921875$ memory_percent‚99.94528293609619( proc.memory.percent‚19.9918270111084 proc.cpu.threads‚17 network.sent‚ 1.5472297e+07 disk.sda.out‚ 301.87109375 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 301.87109375& disk.nvme0n1.out‚239372.6044921875 disk.sda2.out‚0X÷b’ì‚(:‡ ë‘èÔå§Ýx gpu.0.gpu‚61.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚276.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïbݘí‚(: ú‘èÔŠÉx disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.558001e+07 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 302.29296875 disk.sda.out‚ 302.29296875 disk.sda3.out‚ 302.29296875$ disk.nvme0c0n1.in‚24942.33203125( disk.nvme0c0n1.out‚239623.0849609375# disk.dm-0.out‚239623.0849609375 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.out‚ 1.37890625& disk.nvme0n1.out‚239623.0849609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚24942.33203125& proc.memory.availableMB‚ 8.64453125) disk./.usagePercent‚77.98635067158354 disk.sda2.out‚0$ memory_percent‚99.94723796844482 proc.memory.rssMB‚3275.75) proc.memory.percent‚19.99359130859375 cpu‚25.209831521666462 network.recv‚ 3.122547e+06% disk./.usageGB‚1162.0529823303223 disk.drbd1260.in‚0" disk.nvme0n1.in‚24942.33203125Æ—Oõî‚(:’ ú‘èÔÕ‹Òx gpu.0.gpu‚77.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚181.381' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.276199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FzCÍ™ï‚(:Ž ‰’èÔùºíx disk.sda2.in‚0% proc.memory.availableMB‚ 8.1484375# proc.memory.rssMB‚ 3276.03515625% disk./.usageGB‚1162.1782112121582( disk.nvme0c0n1.out‚240042.0927734375 disk.dm-0.in‚24942.33203125 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 302.8984375 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 3.127189e+06 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 302.8984375 disk.sda.out‚ 302.8984375& disk.nvme0n1.out‚240042.0927734375$ memory_percent‚99.95026588439941 cpu‚25.20991482109117 proc.cpu.threads‚17# disk.dm-0.out‚240042.0927734375 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0 disk.sda2.out‚0) proc.memory.percent‚19.99533176422119 network.sent‚ 1.5586361e+07) disk./.usagePercent‚77.99475488691758$ disk.nvme0c0n1.in‚24942.33203125 disk.drbd1260.out‚ 1.3828125" disk.nvme0n1.in‚24942.33203125ŒŠèð‚(:… ‰’èÔ‘ÊŠy gpu.0.gpu‚39.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚193.065' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.613 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ôâ:‘ñ‚(:† ˜’èÔÆÛèx disk.sda2.out‚0 disk.sda1.out‚0 network.recv‚ 3.130763e+06" disk.nvme0c0n1.in‚ 25048.265625 disk.dm-1.out‚ 303.3046875 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0& disk.nvme0n1.out‚240387.6435546875# memory_percent‚99.9523401260376& proc.memory.availableMB‚ 7.80859375 disk.dm-0.in‚ 25048.265625 disk.sda1.in‚0 cpu‚25.209949868823823 proc.cpu.threads‚17) disk./.usagePercent‚77.99315842393406% disk./.usageGB‚1162.1544227600098# disk.dm-0.out‚240387.6435546875 disk.dm-1.in‚ 0.00390625 disk.drbd1260.out‚ 1.3828125 disk.nvme0n1.in‚ 25048.265625 proc.memory.rssMB‚ 3276.3125* proc.memory.percent‚19.997024536132812 network.sent‚ 1.5591236e+07( disk.nvme0c0n1.out‚240387.6435546875 disk.sda.out‚ 303.3046875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 303.3046875 disk.sda2.in‚09G Oò‚(:… ˜’èԔ߼z gpu.0.gpu‚1.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚112.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºçg‹ó‚(:€ §’èÔ¼Üx disk.sda3.in‚ 0.00390625 disk.sda1.in‚0 cpu‚25.209836455736625$ disk./.usageGB‚1162.427890777588" disk.nvme0c0n1.in‚ 25048.265625) disk.nvme0c0n1.out‚240769.45263671875 disk.sda.out‚303.625 disk.sda3.out‚303.625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 25048.265625 disk.sda2.in‚0$ disk.dm-0.out‚240769.45263671875 disk.drbd1260.out‚ 1.38671875 disk.sda2.out‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 7.93359375* proc.memory.percent‚19.997882843017578 proc.cpu.threads‚17 network.sent‚ 1.5597656e+07 network.recv‚ 3.135777e+06 disk.dm-0.in‚ 25048.265625 disk.dm-1.out‚303.625' disk.nvme0n1.out‚240769.45263671875$ memory_percent‚99.95157718658447! proc.memory.rssMB‚ 3276.453125) disk./.usagePercent‚78.01151109204868 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.003906254œTô‚(:„ §’èÔçåy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚34.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿìÕœõ‚(:‘ ¶’èԾČy gpu.0.gpu‚44.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚184.073' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.814600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}O_™ö‚(:Ž ¶’èÔ½‰{ disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚99.95472431182861) disk./.usagePercent‚78.01151621219937" disk.nvme0c0n1.in‚ 25048.265625 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 304.0546875' disk.nvme0n1.out‚241115.12548828125* proc.memory.percent‚20.000100135803223 proc.cpu.threads‚17 network.recv‚ 3.139291e+06% disk./.usageGB‚1162.4279670715332& proc.memory.availableMB‚ 7.41796875# proc.memory.rssMB‚ 3276.81640625 cpu‚25.20990871867066) disk.nvme0c0n1.out‚241115.12548828125 disk.dm-0.in‚ 25048.265625$ disk.dm-0.out‚241115.12548828125 disk.dm-1.out‚ 304.0546875 disk.sda.out‚ 304.0546875 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.5602554e+07 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.out‚ 1.38671875 disk.nvme0n1.in‚ 25048.265625;ÿñ(4÷/ _wandbruntime‚10010  _runtime‚10010,§/Èø _step‚452 & lr-AdamW/pg1‚0.0004277518476915465 & lr-AdamW/pg2‚0.0004277518476915465 trainer/global_step‚5549 % _timestamp‚1.7884798034088762e+09 _runtime‚10010.164834898ÚéÍùÇ % _timestamp‚1.7884798034088762e+09 _runtime‚10010.164834898 _step‚452 & lr-AdamW/pg1‚0.0004277518476915465 & lr-AdamW/pg2‚0.0004277518476915465 trainer/global_step‚5549Ä»M”›4ú/ _wandbruntime‚10010  _runtime‚10010®QË(ûP H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138942556 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859581106761136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚894157409175040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13496291454450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894153772122624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚880637421317120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13516377197056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚453 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚894153772122624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚797099206700032 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117285021 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124707694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚873073764124934 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117285015 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138916216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚880637421317120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚797099155496960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117285015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256227577 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859581081333168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13519987857920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18605222425088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚880637394925568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74179685 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13519987857920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13516377197056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚815704429125120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50528014 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10010.166345676 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117285021 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138942556 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13492682791766 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894157409175040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚815704305984000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884798034099884e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138916216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256201231 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚880637394925568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894153772122624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚880637421317120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13516377197056 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚453 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚894153772122624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚797099206700032 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117285021 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124707694 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚873073764124934 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117285015 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138916216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚880637421317120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚797099155496960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117285015 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256227577 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859581081333168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13519987857920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18605222425088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚880637394925568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74179685 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13519987857920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13516377197056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚815704429125120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50528014 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10010.166345676 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117285021 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138942556 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13492682791766 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894157409175040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚815704305984000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 % _timestamp‚1.7884798034099884e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138916216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256201231 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚880637394925568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138942556 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859581106761136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424ʼn.bh4ý/ _wandbruntime‚10011  _runtime‚10011lÔþ—N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256272594 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚815847049931264 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256246222 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚880791198876160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117305445 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138967149 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894313597229056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18608478820352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚880791198876160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚873226499669356 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124729185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256272594 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13522398352896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256246222 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚797238497089024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884798045119736e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13498697757944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894313597229056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚859731410574304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13518787678720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138940783 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0™©("'ÿœN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894313597229056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚859731410574304 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13518787678720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138940783 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256272594 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚815847049931264 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256246222 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚880791198876160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117305445 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138967149 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894313597229056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18608478820352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚880791198876160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚873226499669356 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124729185 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256272594 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13522398352896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256246222 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚797238497089024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884798045119736e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13498697757944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F[ÔÓZÔ 0loss/FBt4/assignment_loss‚0.2183190346113406 _step‚455 .loss/FRt2/detection_loss‚1.991187722637103 0loss/FRt2/assignment_loss‚2.9215949326753616 epoch‚4 0loss/FRt1/assignment_loss‚2.1404682139852156 /loss/FRt3/assignment_loss‚4.650116523106893 % _timestamp‚1.7884798045144386e+09 _runtime‚10011.270148135 0loss/FRt4/assignment_loss‚3.7064420953392982 1loss/SRqqt1/assignment_loss‚1.020709333524108 2loss/SRqqt2/assignment_loss‚2.0153560861945152 1loss/SRqqt3/detection_loss‚2.5995769716542343 2loss/SRqqt4/assignment_loss‚1.4720885787597473 0loss/FBt3/assignment_loss‚0.4755028891816307 trainer/global_step‚5549 0loss/FBt1/assignment_loss‚0.5188104665527741 1loss/FBt2/assignment_loss‚0.23761482947421086 .loss/FRt1/detection_loss‚1.493481913280027 1loss/SRqqt1/detection_loss‚0.9692847473621369 .loss/FBt3/detection_loss‚2.566163331270218 /loss/FBt4/detection_loss‚4.6383772730827335 /loss/FRt3/detection_loss‚2.9333595109708384 1loss/SRqqt4/detection_loss‚2.1447849370100918 /loss/FBt2/detection_loss‚0.6573658692924416 %loss/total_loss‚49.57060007108942 2loss/SRqqt3/assignment_loss‚1.8456160487799809 .loss/FRt4/detection_loss‚2.972265623509884 0loss/SRqqt2/detection_loss‚2.997122883796692 /loss/FBt1/detection_loss‚2.3849902550379434fÄ`p_‚Ù 2loss/SRqqt4/assignment_loss‚1.4720885787597473 0loss/FBt3/assignment_loss‚0.4755028891816307 trainer/global_step‚5549 0loss/FBt1/assignment_loss‚0.5188104665527741 1loss/FBt2/assignment_loss‚0.23761482947421086 .loss/FRt1/detection_loss‚1.493481913280027 1loss/SRqqt1/detection_loss‚0.9692847473621369 .loss/FBt3/detection_loss‚2.566163331270218 /loss/FBt4/detection_loss‚4.6383772730827335 /loss/FRt3/detection_loss‚2.9333595109708384 1loss/SRqqt4/detection_loss‚2.1447849370100918 /loss/FBt2/detection_loss‚0.6573658692924416 %loss/total_loss‚49.57060007108942 2loss/SRqqt3/assignment_loss‚1.8456160487799809 .loss/FRt4/detection_loss‚2.972265623509884 0loss/SRqqt2/detection_loss‚2.997122883796692 /loss/FBt1/detection_loss‚2.3849902550379434 0loss/FBt4/assignment_loss‚0.2183190346113406 _step‚455 .loss/FRt2/detection_loss‚1.991187722637103 0loss/FRt2/assignment_loss‚2.9215949326753616 epoch‚4 0loss/FRt1/assignment_loss‚2.1404682139852156 /loss/FRt3/assignment_loss‚4.650116523106893 % _timestamp‚1.7884798045144386e+09 _runtime‚10011.270148135 0loss/FRt4/assignment_loss‚3.7064420953392982 1loss/SRqqt1/assignment_loss‚1.020709333524108 2loss/SRqqt2/assignment_loss‚2.0153560861945152 1loss/SRqqt3/detection_loss‚2.5995769716542343ÇË aÇ’ƒ‚(:‡ Å’èÔùĈy gpu.0.gpu‚93.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚168.573' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿h€N’„‚(:‡ Å’èÔŠãìz% disk./.usageGB‚1162.4280700683594 disk.dm-0.in‚ 25048.265625 disk.sda.out‚ 304.46875 disk.sda3.out‚ 304.46875 disk.drbd1260.out‚1.390625 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚99.95815753936768$ disk.dm-0.out‚241366.27099609375* proc.memory.percent‚20.001864433288574 cpu‚25.209969157234838 proc.cpu.threads‚17 network.sent‚ 1.5608793e+07 disk.dm-1.out‚ 304.46875 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0' disk.nvme0n1.out‚241366.27099609375 disk.sda1.out‚0& proc.memory.availableMB‚ 6.85546875# proc.memory.rssMB‚ 3277.10546875 network.recv‚ 3.143973e+06) disk./.usagePercent‚78.01152312440279" disk.nvme0c0n1.in‚ 25048.265625) disk.nvme0c0n1.out‚241366.27099609375 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.nvme0n1.in‚ 25048.265625 disk.sda2.in‚0#í´lœ…‚(:‘ Ô’èÔ´ûy disk.sda.out‚ 304.8828125 disk.sda3.in‚ 0.00390625# proc.memory.rssMB‚ 3277.42578125 network.recv‚ 3.149156e+06 disk.dm-0.in‚25204.66015625" disk.nvme0n1.in‚25204.66015625' disk.nvme0n1.out‚241616.81396484375 disk.sda2.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 6.48828125 proc.cpu.threads‚17 network.sent‚ 1.5716409e+07) disk./.usagePercent‚78.01152312440279 disk.sda3.out‚ 304.8828125 disk.drbd1260.in‚0 disk.drbd1260.out‚1.390625 disk.sda2.out‚0 disk.sda1.out‚0 cpu‚25.2100329974584) disk.nvme0c0n1.out‚241616.81396484375 disk.sda.in‚ 0.00390625$ memory_percent‚99.96039867401123* proc.memory.percent‚20.003819465637207% disk./.usageGB‚1162.4280700683594$ disk.nvme0c0n1.in‚25204.66015625$ disk.dm-0.out‚241616.81396484375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 304.8828125 ñ¥‘†‚(:† Ô’èÔ‰z gpu.0.gpu‚21.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚123.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôZ ´œ‡‚(:‘ ã’èÔ £¡}$ disk.dm-0.out‚241962.99951171875 disk.sda.out‚ 305.2109375 disk.sda3.out‚ 305.2109375 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚25204.66015625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 305.2109375 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.04507777590813 disk.sda3.in‚ 0.00390625" disk.nvme0n1.in‚25204.66015625% proc.memory.availableMB‚ 6.8828125) proc.memory.percent‚20.00565528869629 proc.cpu.threads‚17 network.sent‚ 1.5722868e+07) disk.nvme0c0n1.out‚241962.99951171875 disk.sda.in‚ 0.00390625 disk.drbd1260.out‚ 1.39453125' disk.nvme0n1.out‚241962.99951171875# memory_percent‚99.9579906463623" proc.memory.rssMB‚ 3277.7265625 cpu‚25.210099656126545 network.recv‚ 3.153979e+06% disk./.usageGB‚1162.9280586242676 disk.dm-0.in‚25204.66015625Ý25á’ˆ‚(:‡ ã’èÔÂÖ±} gpu.0.gpu‚78.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚205.138' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.0276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+@§ž‰‚(:“ ò’èÔèþöw network.sent‚ 1.572784e+07 network.recv‚ 3.157155e+06$ disk.dm-0.out‚242381.66357421875 disk.sda3.out‚ 305.75390625 disk.drbd1260.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3278.0234375 cpu‚25.21016157356172) disk./.usagePercent‚78.04507777590813% disk./.usageGB‚1162.9280586242676 disk.dm-0.in‚25204.66015625 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚25204.66015625$ memory_percent‚99.96132850646973) proc.memory.percent‚20.00746726989746 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.39453125 disk.dm-1.out‚ 305.75390625 disk.sda.out‚ 305.75390625" disk.nvme0n1.in‚25204.66015625' disk.nvme0n1.out‚242381.66357421875 disk.sda1.out‚0) disk.nvme0c0n1.out‚242381.66357421875% proc.memory.availableMB‚ 6.3359375ìl©Ù‘Š‚(:† ò’èÔ’¯“y gpu.0.gpu‚22.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚209.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.8744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éùÔM‹‚(:’ “èÔ®çîx disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 306.14453125 disk.drbd1260.out‚ 1.3984375' disk.nvme0n1.out‚242992.86767578125# proc.memory.rssMB‚ 3278.19921875 network.recv‚ 3.162047e+06) disk./.usagePercent‚78.04507649587046$ disk.dm-0.out‚242992.86767578125 disk.dm-1.out‚ 306.14453125 disk.sda1.out‚0* proc.memory.percent‚20.008540153503418 cpu‚25.21024009375554" disk.nvme0n1.in‚25204.66015625 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.availableMB‚6.15625 proc.cpu.threads‚17 network.sent‚ 1.5733888e+07 disk.dm-0.in‚25204.66015625 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚25204.66015625) disk.nvme0c0n1.out‚242992.86767578125# memory_percent‚99.9624252319336% disk./.usageGB‚1162.9280395507812 disk.sda.in‚ 0.00390625 disk.sda.out‚ 306.14453125ì}$´›Œ‚(: “èÔìØ™y gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚150.495' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.099000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯W<”‚(:‰ “èÔÈî‡z' disk.nvme0n1.out‚243146.10986328125 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 306.5546875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 306.5546875 disk.nvme0n1.in‚ 25358.90625 disk.sda1.in‚0$ disk.dm-0.out‚243146.10986328125 disk.sda.out‚ 306.5546875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.3984375# memory_percent‚99.9657392501831& proc.memory.availableMB‚ 5.61328125) disk.nvme0c0n1.out‚243146.10986328125 disk.dm-0.in‚ 25358.90625 network.sent‚ 1.5738764e+07 network.recv‚ 3.165244e+06 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 25358.90625 disk.sda.in‚ 0.00390625# proc.memory.rssMB‚ 3278.49609375 disk.sda2.out‚0) proc.memory.percent‚20.01035213470459 cpu‚25.210292453573942 proc.cpu.threads‚17) disk./.usagePercent‚78.04507803191566% disk./.usageGB‚1162.9280624389648máÊAŽ‚(:’ “èÔ—¾®z gpu.0.gpu‚54.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚171.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.397000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÕ¼4/ _wandbruntime‚10102  _runtime‚10102¬spÉÉà ' lr-AdamW/pg1‚0.00042327179762628037 ' lr-AdamW/pg2‚0.00042327179762628037 trainer/global_step‚5599 $ _timestamp‚1.788479896013419e+09 _runtime‚10102.769713512 _step‚456ºIzÑÎ‘È ' lr-AdamW/pg1‚0.00042327179762628037 ' lr-AdamW/pg2‚0.00042327179762628037 trainer/global_step‚5599 $ _timestamp‚1.788479896013419e+09 _runtime‚10102.769713512 _step‚456Ⱦf—4’/ _wandbruntime‚10102  _runtime‚10102½§ÖÂ6“P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125789932 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118306215 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 _runtime‚10102.771702802 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13636915694080 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140147918 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚804063674946560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚804063726149632 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140174258 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚901966825624064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚880710554937242 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 trainer/global_step‚5599 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258480479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69840896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867097543389968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚901963188571648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118306221 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140147918 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888326299269120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74819935 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚822835347078656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚901963188571648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888326272877568 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118306215 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140174258 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50970708 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18771620929024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚901966825624064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118306221 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18771551088128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13640526354944 % _timestamp‚1.7884798960157747e+09 _step‚457 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888326299269120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74819930 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50970002 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532¥‘ôE(”’P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13640526354944 % _timestamp‚1.7884798960157747e+09 _step‚457 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚888326299269120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74819930 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50970002 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125789932 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118306215 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 _runtime‚10102.771702802 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13636915694080 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140147918 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚804063674946560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚804063726149632 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140174258 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚901966825624064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚880710554937242 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 trainer/global_step‚5599 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258480479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69840896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867097543389968 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚901963188571648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118306221 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140147918 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888326299269120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74819935 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚822835347078656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚901963188571648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888326272877568 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118306215 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140174258 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50970708 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18771620929024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚901966825624064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118306221 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18771551088128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÉŒaé4•/ _wandbruntime‚10103  _runtime‚10103‰J÷"'–—N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚804202965335552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258525496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13619026513452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140198851 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74832735 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50979531 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125812271 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚880866924572524 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258499124 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18774869982208 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867247898059072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902123013678080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69827584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚880863290481664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118326639 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258499124 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118326645 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888480076828160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888480050436608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13639326175744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125811599 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258525496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74832740 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50978864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140198851 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13642936849920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18774939809792 trainer/global_step‚5599 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888480076828160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚804203016538624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13642936849920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888480050436608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867247898059072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902123013678080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚69827584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚880863290481664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118326639 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258499124 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118326645 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888480076828160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888480050436608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13639326175744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125811599 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258525496 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74832740 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50978864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140198851 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13642936849920 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18774939809792 trainer/global_step‚5599 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888480076828160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚804203016538624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13642936849920 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888480050436608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 _ ÜZ}‹ KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚804202965335552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258525496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13619026513452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140198851 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74832735 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50979531 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125812271 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚672 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚880866924572524 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258499124 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18774869982208 BDöÏ©É _step‚460 ' lr-AdamW/pg1‚0.00041877146085303907 ' lr-AdamW/pg2‚0.00041877146085303907 trainer/global_step‚5649 % _timestamp‚1.7884799895700028e+09 _runtime‚10196.326747602ÌEÞô©4ª/ _wandbruntime‚10196  _runtime‚10196]k*(«P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260733381 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51410781 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚742 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18937797835264 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260707035 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126870966 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874614005446768 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141379620 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚909776242073088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75460180 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141405960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚896015177221120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚896015150829568 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚896015150829568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚896015177221120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75460185 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126870219 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141405960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909772605020672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13733340302782 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119327421 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _step‚461 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚10196.331674299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13736948965466 trainer/global_step‚5649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚811028245599232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚888350979840202 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141379620 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18937871870464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚829965992231424 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13757454191104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13761064851968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚888347345749550 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 % _timestamp‚1.7884799895736897e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚909772605020672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126870219 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141405960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909772605020672 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13733340302782 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119327421 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _step‚461 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚10196.331674299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13736948965466 trainer/global_step‚5649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚Ö:opA26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚811028245599232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚888350979840202 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141379620 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚747 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18937871870464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚829965992231424ÍÈ/¨†4­/ _wandbruntime‚10198  _runtime‚10198?¾ù3'®—N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚874764360115872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126891782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51418797 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚909932430127104 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260778398 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚896168954780160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13763475346944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚830108756121600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13759864672768 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18941146111488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884799918265777e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119347845 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119347845 trainer/global_step‚5649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚896168928388608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909928793061376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18941220133376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 _step‚462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909932430127104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚10198.583939587 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13735746606068 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13763475346944 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896168928388608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119347839 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119347839 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260752026 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141404187 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚888500081293972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260778398 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141404187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141430553 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚811167535988224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75472990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚714 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚830108630896640«Ê$Ð"'¯œN K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119347839 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260752026 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141404187 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚888500081293972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260778398 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141404187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141430553 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚811167535988224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75472990 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚714 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚830108630896640 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚874764360115872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126891782 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51418797 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚909932430127104 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260778398 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚896168954780160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13763475346944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚830108756121600 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13759864672768 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18941146111488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884799918265777e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119347845 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119347845 trainer/global_step‚5649 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚896168928388608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909928793061376 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18941220133376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 _step‚462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909932430127104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚719 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 _runtime‚10198.583939587 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13735746606068 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13763475346944 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚896168928388608 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119347839γœ9 4°/ _wandbruntime‚10198  _runtime‚10198ÀSµÍ\±Ö 0loss/FBt3/assignment_loss‚0.2909769430062148 /loss/FBt1/detection_loss‚2.1344470931933475 trainer/global_step‚5649 0loss/FBt2/assignment_loss‚0.2526614790900914 /loss/FRt2/detection_loss‚1.9557881791209966 .loss/FRt3/detection_loss‚3.134551698511297 .loss/FBt3/detection_loss‚2.642870306968689 %loss/total_loss‚46.02757496317567 0loss/FRt2/assignment_loss‚3.1373530448528757 .loss/FRt3/assignment_loss‚4.11943489854986 /loss/FRt1/detection_loss‚1.6625803397918228 /loss/FBt2/detection_loss‚0.9628473445482086 _runtime‚10198.584712905 2loss/SRqqt2/assignment_loss‚2.2807269215583803 2loss/SRqqt3/assignment_loss‚1.9563873226719874 /loss/FRt4/detection_loss‚2.6276906842277166 1loss/SRqqt1/detection_loss‚1.1504820583484554 0loss/SRqqt2/detection_loss‚2.992824463049571 0loss/FRt1/assignment_loss‚1.5340794535244213 0loss/FRt4/assignment_loss‚2.9602434975760326 epoch‚4 _step‚463 1loss/SRqqt3/detection_loss‚2.3146493004428015 /loss/FBt4/detection_loss‚2.5331243587576826 1loss/FBt4/assignment_loss‚0.41989273086864676 2loss/SRqqt1/assignment_loss‚0.9653616808194282 1loss/SRqqt4/assignment_loss‚1.644560132175684 1loss/FBt1/assignment_loss‚0.10649257950177134 1loss/SRqqt4/detection_loss‚2.2475484520196916 % _timestamp‚1.7884799918289392e+09œºBa²Û 1loss/FBt4/assignment_loss‚0.41989273086864676 2loss/SRqqt1/assignment_loss‚0.9653616808194282 1loss/SRqqt4/assignment_loss‚1.644560132175684 1loss/FBt1/assignment_loss‚0.10649257950177134 1loss/SRqqt4/detection_loss‚2.2475484520196916 % _timestamp‚1.7884799918289392e+09 0loss/FBt3/assignment_loss‚0.2909769430062148 /loss/FBt1/detection_loss‚2.1344470931933475 trainer/global_step‚5649 0loss/FBt2/assignment_loss‚0.2526614790900914 /loss/FRt2/detection_loss‚1.9557881791209966 .loss/FRt3/detection_loss‚3.134551698511297 .loss/FBt3/detection_loss‚2.642870306968689 %loss/total_loss‚46.02757496317567 0loss/FRt2/assignment_loss‚3.1373530448528757 .loss/FRt3/assignment_loss‚4.11943489854986 /loss/FRt1/detection_loss‚1.6625803397918228 /loss/FBt2/detection_loss‚0.9628473445482086 _runtime‚10198.584712905 2loss/SRqqt2/assignment_loss‚2.2807269215583803 2loss/SRqqt3/assignment_loss‚1.9563873226719874 /loss/FRt4/detection_loss‚2.6276906842277166 1loss/SRqqt1/detection_loss‚1.1504820583484554 0loss/SRqqt2/detection_loss‚2.992824463049571 0loss/FRt1/assignment_loss‚1.5340794535244213 0loss/FRt4/assignment_loss‚2.9602434975760326 epoch‚4 _step‚463 1loss/SRqqt3/detection_loss‚2.3146493004428015 /loss/FBt4/detection_loss‚2.5331243587576826Ïùc:…’³‚(:‡ ù“èÔú¢«y gpu.0.gpu‚34.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.117' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6234 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R m¼ ´‚(:• ù“èÔ Òñy network.recv‚ 3.196431e+06 disk.sda2.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚25494.69140625 disk.sda.out‚ 310.19140625 disk.drbd1260.in‚0$ proc.memory.availableMB‚7.515625 network.sent‚ 1.5881461e+07 disk.sda2.in‚0) disk.nvme0c0n1.out‚245569.72607421875 disk.dm-0.in‚25494.69140625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 310.19140625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625) disk./.usagePercent‚78.04351740998693% disk./.usageGB‚1162.9048080444336$ disk.dm-0.out‚245569.72607421875 disk.sda3.out‚ 310.19140625" disk.nvme0n1.in‚25494.69140625$ memory_percent‚99.95412826538086* proc.memory.percent‚20.022273063659668 cpu‚25.21069824530844 disk.sda1.out‚0 disk.drbd1260.out‚ 1.41796875' disk.nvme0n1.out‚245569.72607421875# proc.memory.rssMB‚ 3280.44921875 proc.cpu.threads‚17L¤‡H’µ‚(:‡ ˆ”èÔµ’åx gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚240.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý‹ñ¹›¶‚(: ˆ”èÔ±ôêy$ memory_percent‚99.95729923248291) disk./.usagePercent‚78.04509672046565% disk./.usageGB‚1162.9283409118652 disk.sda.out‚ 310.6484375 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚245819.2822265625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625& disk.nvme0n1.out‚245819.2822265625 disk.dm-0.in‚25494.69140625" proc.memory.rssMB‚ 3280.7265625) proc.memory.percent‚20.02396583557129 cpu‚25.210755691730085 proc.cpu.threads‚17 network.sent‚ 1.5989061e+07 disk.sda3.out‚ 310.6484375 disk.drbd1260.out‚ 1.41796875 disk.sda2.in‚0( disk.nvme0c0n1.out‚245819.2822265625& proc.memory.availableMB‚ 6.99609375 network.recv‚ 3.201635e+06 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 310.6484375" disk.nvme0n1.in‚25494.69140625$ disk.nvme0c0n1.in‚25494.69140625Wç”Ý’·‚(:‡ —”èÔ§¬áx gpu.0.gpu‚54.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03WÇ•¸‚(:Š —”èÔ¯¢…z) disk./.usagePercent‚78.04372324004446 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.42578125 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 25661.1484375 disk.dm-0.in‚ 25661.1484375# disk.dm-0.out‚246410.3291015625 network.sent‚ 1.5995427e+07 network.recv‚ 3.205965e+06% disk./.usageGB‚1162.9078750610352 disk.dm-1.out‚ 311.328125) proc.memory.percent‚20.02396583557129 cpu‚25.210811669481664 proc.cpu.threads‚17 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625$ memory_percent‚99.95734691619873& proc.memory.availableMB‚ 6.98828125" proc.memory.rssMB‚ 3280.7265625! disk.nvme0n1.in‚ 25661.1484375& disk.nvme0n1.out‚246410.3291015625 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚246410.3291015625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 311.328125 disk.sda3.out‚ 311.328125¬x'%‘¹‚(:† ¦”èÔç“Êy gpu.0.gpu‚35.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚162.471' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý¢NÚ˜º‚(: ¦”èÔé½Åz disk.drbd1260.out‚ 1.4296875 disk.dm-0.in‚ 25661.1484375# disk.dm-0.out‚246801.7548828125 disk.dm-1.in‚ 0.00390625 network.recv‚ 3.209003e+06 disk.sda2.in‚0 disk.dm-1.out‚ 312.15234375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 25661.1484375 network.sent‚ 1.5999444e+07( disk./.usagePercent‚78.0452997344403 disk.sda3.in‚ 0.00390625% proc.memory.availableMB‚ 7.2421875" proc.memory.rssMB‚ 3280.7265625) proc.memory.percent‚20.02396583557129 cpu‚25.210841839112657 disk.sda3.out‚ 312.15234375& disk.nvme0n1.out‚246801.7548828125 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 25661.1484375( disk.nvme0c0n1.out‚246801.7548828125$ memory_percent‚99.95579719543457% disk./.usageGB‚1162.9313659667969 disk.sda.in‚ 0.00390625 disk.sda.out‚ 312.15234375 disk.drbd1260.in‚0«Öx:»‚(:‚ µ”èÔÒ«ëz$ memory_percent‚99.95582103729248( disk./.usagePercent‚77.9676539293429 disk.dm-0.in‚ 25661.1484375 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0% disk.nvme0n1.out‚247256.732421875" proc.memory.rssMB‚ 3280.7265625) proc.memory.percent‚20.02396583557129 network.sent‚ 1.6005635e+07 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 25661.1484375" disk.dm-0.out‚247256.732421875 disk.drbd1260.out‚1.5& proc.memory.availableMB‚ 7.23828125 cpu‚25.210824576192778 network.recv‚ 3.213443e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 312.6015625 disk.sda2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1161.7743873596191' disk.nvme0c0n1.out‚247256.732421875 disk.dm-1.out‚ 312.6015625 disk.sda.out‚ 312.6015625! disk.nvme0n1.in‚ 25661.1484375 disk.sda2.out‚0AZç÷¼‚(:’ µ”èÔçÛÝz gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚273.818' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.763600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿¹sŒ4½/ _wandbruntime‚10274  _runtime‚10274óf6kÊ¾Ä trainer/global_step‚5699 % _timestamp‚1.7884800673167672e+09 _runtime‚10274.072543659 _step‚464 ' lr-AdamW/pg1‚0.00041425177523526067 ' lr-AdamW/pg2‚0.00041425177523526067C”ÿ\Ï¿É trainer/global_step‚5699 % _timestamp‚1.7884800673167672e+09 _runtime‚10274.072543659 _step‚464 ' lr-AdamW/pg1‚0.00041425177523526067 ' lr-AdamW/pg2‚0.00041425177523526067ÐÞÚV4À/ _wandbruntime‚10274  _runtime‚10274 vçÁP eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚817992765048832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19103902740480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917582021469696 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13877992688128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚895984136561858 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51850791 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19103976775680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120348621 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882130492931536 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13853669058290 _step‚465 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚917585658522112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚837096741824512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚837096616586240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚817992713845760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120348615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142611322 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903704055173120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13881603348992 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262986283 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884800673181992e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142637662 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13857277720974 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚903704028781568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120348621 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51851519 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 trainer/global_step‚5699 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _runtime‚10274.073957354 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚903704028781568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚917582021469696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120348615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚917585658522112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262959937 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903704055173120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142637662 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142611322 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13877992688128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127951221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127951954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76100430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76100435 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0£š,=(Â’P @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76100435 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚817992765048832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19103902740480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚917582021469696 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13877992688128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚895984136561858 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51850791 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19103976775680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120348621 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882130492931536 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13853669058290 _step‚465 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚917585658522112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚837096741824512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚837096616586240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚817992713845760 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120348615 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚733 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142611322 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903704055173120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13881603348992 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262986283 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884800673181992e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142637662 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13857277720974 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚903704028781568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120348621 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51851519 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 trainer/global_step‚5699 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _runtime‚10274.073957354 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚903704028781568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚917582021469696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120348615 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚917585658522112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262959937 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903704055173120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142637662 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142611322 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13877992688128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127951221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127951954 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76100430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ÑŸ„¬.†Ã‚(:û Ä”èÔõ£y$ proc.memory.availableMB‚7.203125 network.recv‚ 3.21678e+06% disk./.usageGB‚1161.8693008422852 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 312.984375 disk.sda2.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 25661.1484375) disk./.usagePercent‚77.97402365280234" disk.dm-0.out‚247546.724609375$ memory_percent‚99.95603561401367" proc.memory.rssMB‚ 3280.7265625) proc.memory.percent‚20.02396583557129 proc.cpu.threads‚17 disk.drbd1260.out‚1.5 disk.dm-1.out‚ 312.984375 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 312.984375 disk.sda1.out‚0' disk.nvme0c0n1.out‚247546.724609375 cpu‚25.210623996179358 network.sent‚ 1.60105e+07 disk.dm-0.in‚ 25661.1484375 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 25661.1484375% disk.nvme0n1.out‚247546.724609375 disk.sda2.in‚0$¥p×’Ä‚(:‡ Ä”èÔš¸Ÿy gpu.0.gpu‚95.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚290.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Œ ê4Å/ _wandbruntime‚10275  _runtime‚10275šP†['Æ—N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚818132004234752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚917738209510400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142662255 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚896136872106280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚818132055437824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142635889 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19107307255808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 % _timestamp‚1.7884800683597925e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127972742 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882280822172672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263004928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51860216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120369045 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚837239237468672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13859684024468 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13884013843968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263004928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263031300 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120369039 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13880403169792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚917741846576128 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903857832732160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚903857806340608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142635889 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120369039 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚837239362693632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903857832732160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127973456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76113240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263031300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19107233233920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13856075361576 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263031300 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19107233233920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13856075361576 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19107307255808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 % _timestamp‚1.7884800683597925e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127972742 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882280822172672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263004928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51860216 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120369045 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚837239237468672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13859684024468 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13884013843968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263004928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263031300 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120369039 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13880403169792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚917741846576128 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903857832732160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚903857806340608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142635889 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120369039 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚837239362693632Ò >EY4È/ _wandbruntime‚10275  _runtime‚10275¸#Ð>]É× 0loss/SRqqt2/detection_loss‚2.573885417646832 /loss/FBt2/detection_loss‚0.7376844692177006 /loss/FBt3/detection_loss‚2.7995874285697937 epoch‚4 0loss/FRt3/assignment_loss‚3.5853225529193877 1loss/FBt3/assignment_loss‚0.23646381272867853 1loss/SRqqt1/detection_loss‚0.9809059735698005 /loss/FBt1/detection_loss‚3.0607046683629355 2loss/SRqqt4/assignment_loss‚1.6650370061397552 0loss/FBt4/assignment_loss‚0.2128650897791431 /loss/FRt2/detection_loss‚1.8826995982478063 .loss/FRt4/detection_loss‚2.710056611985871 &loss/total_loss‚45.503299788981906 2loss/SRqqt1/assignment_loss‚1.1722068177739857 0loss/SRqqt3/detection_loss‚2.566004087527593 trainer/global_step‚5699 _runtime‚10275.11896631 /loss/FRt1/assignment_loss‚2.007454065605998 1loss/SRqqt2/assignment_loss‚2.035117147697343 1loss/FBt2/assignment_loss‚0.12211161501199948 1loss/SRqqt4/detection_loss‚2.2090491404136023 % _timestamp‚1.7884800683635046e+09 0loss/FRt2/assignment_loss‚2.7335364167337066 0loss/FRt4/assignment_loss‚3.4720432975075464 2loss/SRqqt3/assignment_loss‚1.7536656230688095 /loss/FRt1/detection_loss‚1.3767194421055269 _step‚467 .loss/FRt3/detection_loss‚2.946571042140325 /loss/FBt4/detection_loss‚2.5244275480508804 1loss/FBt1/assignment_loss‚0.13918091617688333V~˜zbÊÜ /loss/FRt1/assignment_loss‚2.007454065605998 1loss/SRqqt2/assignment_loss‚2.035117147697343 1loss/FBt2/assignment_loss‚0.12211161501199948 1loss/SRqqt4/detection_loss‚2.2090491404136023 % _timestamp‚1.7884800683635046e+09 0loss/FRt2/assignment_loss‚2.7335364167337066 0loss/FRt4/assignment_loss‚3.4720432975075464 2loss/SRqqt3/assignment_loss‚1.7536656230688095 /loss/FRt1/detection_loss‚1.3767194421055269 _step‚467 .loss/FRt3/detection_loss‚2.946571042140325 /loss/FBt4/detection_loss‚2.5244275480508804 1loss/FBt1/assignment_loss‚0.13918091617688333 0loss/SRqqt2/detection_loss‚2.573885417646832 /loss/FBt2/detection_loss‚0.7376844692177006 /loss/FBt3/detection_loss‚2.7995874285697937 epoch‚4 0loss/FRt3/assignment_loss‚3.5853225529193877 1loss/FBt3/assignment_loss‚0.23646381272867853 1loss/SRqqt1/detection_loss‚0.9809059735698005 /loss/FBt1/detection_loss‚3.0607046683629355 2loss/SRqqt4/assignment_loss‚1.6650370061397552 0loss/FBt4/assignment_loss‚0.2128650897791431 /loss/FRt2/detection_loss‚1.8826995982478063 .loss/FRt4/detection_loss‚2.710056611985871 &loss/total_loss‚45.503299788981906 2loss/SRqqt1/assignment_loss‚1.1722068177739857 0loss/SRqqt3/detection_loss‚2.566004087527593 trainer/global_step‚5699 _runtime‚10275.11896631ÓÜlk<Ë‚(:… Ó”èÔí¯¯x network.sent‚ 1.6108267e+07 network.recv‚ 3.22356e+06 disk.sda3.out‚ 313.3046875$ disk.nvme0n1.out‚247762.55078125 disk.sda2.in‚0 cpu‚25.21044473352236) disk./.usagePercent‚77.97402237276467 disk.sda.in‚ 0.00390625 disk.sda2.out‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚247762.55078125" proc.memory.rssMB‚ 3280.7265625) proc.memory.percent‚20.02396583557129 disk.sda.out‚ 313.3046875 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚25783.61328125 disk.dm-0.in‚25783.61328125 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 313.3046875% disk./.usageGB‚1161.8692817687988 disk.drbd1260.out‚1.546875" disk.nvme0n1.in‚25783.61328125! disk.dm-0.out‚247762.55078125$ memory_percent‚99.95617866516113% proc.memory.availableMB‚ 7.1796875 proc.cpu.threads‚17ƘÉMÌ‚(:’ Ó”èÔú¨öx gpu.0.gpu‚84.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚120.384' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.076800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/¯…‡—Í‚(:Œ â”èÔ•³óx disk.drbd1260.out‚1.546875$ memory_percent‚99.95617866516113" proc.memory.rssMB‚ 3280.7265625) proc.memory.percent‚20.02396583557129 network.sent‚ 1.6205199e+07 disk.drbd1260.in‚0% disk.nvme0n1.out‚248028.494140625 disk.sda1.in‚0' disk.nvme0c0n1.out‚248028.494140625 disk.dm-0.in‚25783.61328125 proc.cpu.threads‚17) disk./.usagePercent‚77.97402390880987% disk./.usageGB‚1161.8693046569824 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 313.73046875" disk.nvme0n1.in‚25783.61328125 disk.sda1.out‚0" disk.dm-0.out‚248028.494140625% proc.memory.availableMB‚ 7.1796875 disk.dm-1.out‚ 313.73046875 disk.sda.in‚ 0.00390625 disk.sda.out‚ 313.73046875 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚25783.61328125 cpu‚25.21030676809158 network.recv‚ 3.228606e+06 disk.sda3.in‚ 0.00390625bu¾’΂(:‡ â”èÔ–ìÙy gpu.0.gpu‚96.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚294.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.8044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó—\€›Ï‚(: ñ”èÔ£íëy gpu.0.gpu‚31.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚169.694' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.93879999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öˆáŠÐ‚(:ÿ ñ”èÔþŒÃ{ disk.sda.out‚314.0625 disk.sda3.out‚314.0625 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.out‚ 1.55078125 disk.sda1.out‚0! proc.memory.availableMB‚7.125 network.recv‚ 3.233419e+06$ disk./.usageGB‚1161.869270324707 disk.dm-0.in‚25783.61328125# disk.dm-0.out‚248413.8662109375" disk.nvme0n1.in‚25783.61328125& disk.nvme0n1.out‚248413.8662109375$ memory_percent‚99.95651245117188) proc.memory.percent‚20.02429962158203( disk.nvme0c0n1.out‚248413.8662109375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 3280.78125 cpu‚25.210302780039633 proc.cpu.threads‚17 network.sent‚ 1.6211358e+07) disk./.usagePercent‚77.97402160474206$ disk.nvme0c0n1.in‚25783.61328125 disk.dm-1.out‚314.0625 disk.sda.in‚ 0.00390625¿ìr’Ñ‚(:‡ €•èÔžå¢x gpu.0.gpu‚95.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë_ ç¢Ò‚(:— €•èÔÁƲz& proc.memory.availableMB‚ 6.85546875 network.sent‚ 1.6216266e+07% disk./.usageGB‚1161.7701683044434$ disk.nvme0c0n1.in‚25783.61328125 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.sda2.out‚0$ memory_percent‚99.95815753936768 network.recv‚ 3.23652e+06) disk./.usagePercent‚77.96737078501003$ disk.dm-0.out‚248702.78662109375" disk.nvme0n1.in‚25783.61328125' disk.nvme0n1.out‚248702.78662109375# proc.memory.rssMB‚ 3280.81640625 cpu‚25.210437393484458 proc.cpu.threads‚17 disk.sda3.out‚ 314.57421875 disk.dm-0.in‚25783.61328125 disk.sda.in‚ 0.00390625 disk.sda.out‚ 314.57421875 disk.sda2.in‚0* proc.memory.percent‚20.024514198303223) disk.nvme0c0n1.out‚248702.78662109375 disk.dm-1.out‚ 314.57421875 disk.drbd1260.out‚ 1.55078125 disk.sda1.in‚0 disk.sda1.out‚0Jp{¹4Ó/ _wandbruntime‚10343  _runtime‚10343?4ÈÔ & lr-AdamW/pg1‚0.0004097136826686538 & lr-AdamW/pg2‚0.0004097136826686538 trainer/global_step‚5749 % _timestamp‚1.7884801367831748e+09 _runtime‚10343.539535612 _step‚468ìëlÍÕÇ _step‚468 & lr-AdamW/pg1‚0.0004097136826686538 & lr-AdamW/pg2‚0.0004097136826686538 trainer/global_step‚5749 % _timestamp‚1.7884801367831748e+09 _runtime‚10343.539535612ÔA\í4Ö/ _wandbruntime‚10343  _runtime‚10343W£QŸ(×P WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚10343.541120339 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14002141846016 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143843024 _step‚469 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13998531185152 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121369821 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚903624561464818 trainer/global_step‚5749 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚911392906733568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚925395074971136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265212839 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265239185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 % _timestamp‚1.7884801367852554e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚889646954988336 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚889646929560368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76740680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121369815 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129031127 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚903620927374166 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚911392906733568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚824957233295360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121369815 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143869364 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76740685 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚925395074971136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚925391437918720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚911392933125120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129031844 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚824957284498432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚911392933125120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52290447 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52291159 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19270072717312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143843024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚844227306012672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13973997813798 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143869364 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76740685 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚925395074971136 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚925391437918720 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚911392933125120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129031844 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚824957284498432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚911392933125120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52290447 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52291159 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19270072717312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143843024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚844227306012672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13973997813798 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123141120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265212839 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265239185 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 % _timestamp‚1.7884801367852554e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚889646954988336 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚889646929560368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76740680 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121369815 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Õ ¶æ4Ù/ _wandbruntime‚10344  _runtime‚10344q$G'Ú—N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 _step‚470 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143893957 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚825096574887424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143867591 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265284202 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76753485 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13980012779976 _runtime‚10344.565213738 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14004552340992 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚925551263025152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13976404117084 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121390245 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚911546710684160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52299935 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚911546710684160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121390239 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 % _timestamp‚1.7884801378085544e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14004552340992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19273400585728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚903773662918588 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129053425 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129052743 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265257830 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚911546684292608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265284202 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265257830 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 trainer/global_step‚5749 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚925551263025152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14000941666816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121390245 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚825096523684352 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143867591 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52299258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143867591 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52299258 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143867591 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265284202 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76753485 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13980012779976 _runtime‚10344.565213738 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14004552340992 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚925551263025152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13976404117084 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121390245 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚911546710684160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52299935 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚911546710684160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121390239 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 % _timestamp‚1.7884801378085544e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14004552340992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19273400585728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚903773662918588 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129053425 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129052743 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265257830 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚911546684292608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265284202 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265257830 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 trainer/global_step‚5749ÖV0‘²4Ü/ _wandbruntime‚10344  _runtime‚10344ËKË?`ÝÚ 2loss/SRqqt4/assignment_loss‚1.7835391568831909 1loss/SRqqt1/detection_loss‚1.0714264754039635 /loss/FBt2/detection_loss‚0.7864744085556185 1loss/FBt2/assignment_loss‚0.09943294427241801 0loss/SRqqt4/detection_loss‚2.183055023352305 /loss/FBt4/detection_loss‚2.3787701196140714 &loss/total_loss‚44.701563190983734 % _timestamp‚1.7884801378109717e+09 0loss/FBt1/assignment_loss‚0.1557718151137427 2loss/SRqqt2/assignment_loss‚1.4800415322894143 2loss/SRqqt3/assignment_loss‚1.4714635820551352 epoch‚4 1loss/FBt4/assignment_loss‚0.20446027260848293 /loss/FRt2/detection_loss‚1.8711374902299471 /loss/FRt4/detection_loss‚3.2287253279771124 1loss/SRqqt2/detection_loss‚2.7643641318593706 1loss/SRqqt3/detection_loss‚2.8192436288703573 -loss/FBt3/detection_loss‚2.29878166142632 0loss/FRt2/assignment_loss‚2.5739980157540767 /loss/FRt3/detection_loss‚3.0246228686085455 /loss/FBt1/detection_loss‚2.4211779634157815 _step‚471 0loss/FRt1/assignment_loss‚1.8615797416108553 trainer/global_step‚5749 _runtime‚10344.566442615 0loss/FRt3/assignment_loss‚4.1419713055645975 2loss/SRqqt1/assignment_loss‚1.2072807563901213 1loss/FBt3/assignment_loss‚0.15199359858353786 /loss/FRt1/detection_loss‚1.7903607764370075 /loss/FRt4/assignment_loss‚2.931890594107764ŇË8eÞß /loss/FRt1/detection_loss‚1.7903607764370075 /loss/FRt4/assignment_loss‚2.931890594107764 2loss/SRqqt4/assignment_loss‚1.7835391568831909 1loss/SRqqt1/detection_loss‚1.0714264754039635 /loss/FBt2/detection_loss‚0.7864744085556185 1loss/FBt2/assignment_loss‚0.09943294427241801 0loss/SRqqt4/detection_loss‚2.183055023352305 /loss/FBt4/detection_loss‚2.3787701196140714 &loss/total_loss‚44.701563190983734 % _timestamp‚1.7884801378109717e+09 0loss/FBt1/assignment_loss‚0.1557718151137427 2loss/SRqqt2/assignment_loss‚1.4800415322894143 2loss/SRqqt3/assignment_loss‚1.4714635820551352 epoch‚4 1loss/FBt4/assignment_loss‚0.20446027260848293 /loss/FRt2/detection_loss‚1.8711374902299471 /loss/FRt4/detection_loss‚3.2287253279771124 1loss/SRqqt2/detection_loss‚2.7643641318593706 1loss/SRqqt3/detection_loss‚2.8192436288703573 -loss/FBt3/detection_loss‚2.29878166142632 0loss/FRt2/assignment_loss‚2.5739980157540767 /loss/FRt3/detection_loss‚3.0246228686085455 /loss/FBt1/detection_loss‚2.4211779634157815 _step‚471 0loss/FRt1/assignment_loss‚1.8615797416108553 trainer/global_step‚5749 _runtime‚10344.566442615 0loss/FRt3/assignment_loss‚4.1419713055645975 2loss/SRqqt1/assignment_loss‚1.2072807563901213 1loss/FBt3/assignment_loss‚0.15199359858353786×úx¤º’ß‚(:‡ •èÔ”ÞÝv gpu.0.gpu‚59.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚270.738' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘+‹0‡à‚(:ü •èÔ³èw# proc.memory.rssMB‚ 3280.81640625 network.sent‚ 1.6222726e+07( disk./.usagePercent‚77.9673039670436% disk.nvme0c0n1.out‚249015.9453125 disk.sda1.out‚0* proc.memory.percent‚20.024514198303223 cpu‚25.21024385430649 proc.cpu.threads‚17 disk.dm-0.in‚ 25916.9375 disk.dm-0.out‚249015.9453125 disk.sda.in‚ 0.00390625 disk.sda.out‚ 314.88671875 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 25916.9375 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 3.241338e+06 disk.nvme0c0n1.in‚ 25916.9375 disk.dm-1.out‚ 314.88671875 disk.sda3.in‚ 0.00390625 disk.drbd1260.out‚ 1.5546875# disk.nvme0n1.out‚249015.9453125 disk.sda1.in‚0& proc.memory.availableMB‚ 6.80078125$ disk./.usageGB‚1161.769172668457 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 314.88671875$ memory_percent‚99.958491325378421±’á‚(:‡ ž•èÔ’Ÿ§w gpu.0.gpu‚67.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚47.0 gpu.0.powerWatts‚288.143' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^kå]}â‚(:ò ž•èÔÿ¹äw disk.sda.in‚ 0.00390625 disk.sda.out‚ 315.34375# proc.memory.rssMB‚ 3280.81640625* proc.memory.percent‚20.024514198303223 cpu‚25.210085982455706% disk.nvme0c0n1.out‚249191.9453125% disk./.usageGB‚1161.9270057678223 disk.dm-1.out‚ 315.34375 disk.sda3.in‚ 0.00390625$ memory_percent‚99.95765686035156" proc.memory.availableMB‚6.9375 network.recv‚ 3.246176e+06) disk./.usagePercent‚77.97789627877279 disk.nvme0c0n1.in‚ 25916.9375 disk.dm-0.in‚ 25916.9375 disk.sda3.out‚ 315.34375 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 25916.9375# disk.nvme0n1.out‚249191.9453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚249191.9453125 network.sent‚ 1.6330355e+07 disk.drbd1260.out‚ 1.5546875 disk.dm-1.in‚ 0.00390625÷æyˆã‚(:ý ­•èÔßÞz disk.sda1.in‚0$ memory_percent‚99.95832443237305 proc.cpu.threads‚17) disk./.usagePercent‚77.97788271037346# disk.dm-0.out‚249633.8408203125 disk.dm-1.out‚ 315.65625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 315.65625 disk.nvme0n1.in‚ 25916.9375 disk.sda2.out‚0$ proc.memory.availableMB‚6.828125 network.sent‚ 1.6338446e+07 network.recv‚ 3.250897e+06 disk.dm-0.in‚ 25916.9375 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3280.81640625% disk./.usageGB‚1161.9268035888672( disk.nvme0c0n1.out‚249633.8408203125 disk.sda3.out‚ 315.65625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.55859375* proc.memory.percent‚20.024514198303223 cpu‚25.21019990700832 disk.nvme0c0n1.in‚ 25916.9375& disk.nvme0n1.out‚249633.8408203125d¨È›ä‚(: ­•èÔÑ‚™{ gpu.0.gpu‚47.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚210.473' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚42.09460000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xêøj’å‚(:‡ ¼•èÔ¼Ÿäw gpu.0.gpu‚66.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚267.494' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LSæ‚(:ù ¼•èÔÖÐéx* proc.memory.percent‚20.024514198303223 proc.cpu.threads‚17 network.recv‚ 3.254141e+06( disk./.usagePercent‚77.9782022077762% disk./.usageGB‚1161.9315643310547 disk.sda1.out‚0 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda.out‚316.1875 disk.drbd1260.in‚0 network.sent‚ 1.6341608e+07 disk.nvme0c0n1.in‚ 25916.9375( disk.nvme0c0n1.out‚250090.7666015625# disk.dm-0.out‚250090.7666015625 disk.nvme0n1.in‚ 25916.9375& disí™×­~k.nvme0n1.out‚250090.7666015625# proc.memory.availableMB‚6.40625# proc.memory.rssMB‚ 3280.81640625 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 25916.9375 disk.drbd1260.out‚ 1.55859375 cpu‚25.210123335242855 disk.sda2.out‚0 disk.dm-1.out‚316.1875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚316.1875$ memory_percent‚99.96089935302734Óq[œç‚(:‘ Ë•èÔÄÏÛx gpu.0.gpu‚12.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚126.786' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.357200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÂì“è‚(:ˆ Ë•èÔ­ã‘y* proc.memory.percent‚20.024514198303223 cpu‚25.210131244514905 network.recv‚ 3.258907e+06) disk./.usagePercent‚77.97819606359539% disk./.usageGB‚1161.9314727783203$ disk.dm-0.out‚250288.54833984375 disk.sda.in‚ 0.00390625 disk.sda.out‚ 316.515625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 316.515625 network.sent‚ 1.6348063e+07! disk.nvme0n1.in‚ 26011.0546875 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 26011.0546875 disk.drbd1260.out‚1.5625$ memory_percent‚99.96013641357422# proc.memory.availableMB‚6.53125 proc.cpu.threads‚17' disk.nvme0n1.out‚250288.54833984375 disk.sda2.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚250288.54833984375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 316.515625 disk.drbd1260.in‚0# proc.memory.rssMB‚ 3280.81640625 disk.sda1.out‚0 disk.dm-0.in‚ 26011.0546875‰°cš4é/ _wandbruntime‚10420  _runtime‚10420 ÑnÊêÄ trainer/global_step‚5799 % _timestamp‚1.7884802138754814e+09 _runtime‚10420.631548964 _step‚472 ' lr-AdamW/pg1‚0.00040515812888490733 ' lr-AdamW/pg2‚0.00040515812888490733žMÿÏëÉ % _timestamp‚1.7884802138754814e+09 _runtime‚10420.631548964 _step‚472 ' lr-AdamW/pg1‚0.00040515812888490733 ' lr-AdamW/pg2‚0.00040515812888490733 trainer/global_step‚5799ØÔDÎ4ì/ _wandbruntime‚10420  _runtime‚10420œ!°–(íP YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚919081784685568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14122680343040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122391015 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14097935231990 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚473 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145074726 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14122680343040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52731526 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚851358288893440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚897163417045136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚897163391617168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52730851 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14094326569306 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122391021 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚831921752744960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69840896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚911257718186474 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 % _timestamp‚1.7884802138773272e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚933200854367744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚919081811077120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 _runtime‚10420.633194877 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122391015 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145101066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚831921803948032 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267465741 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚851358167849472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267492087 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77380930 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14119069682176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933200854367744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130112461 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121043968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚911261352277126 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122391021 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚919081784685568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19436415104512 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145074726 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719¯¢Ë(î’P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145074726 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14122680343040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52731526 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚851358288893440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚897163417045136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚897163391617168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52730851 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14094326569306 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122391021 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚831921752744960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚675 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69840896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚911257718186474 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 % _timestamp‚1.7884802138773272e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚933200854367744 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚919081811077120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 _runtime‚10420.633194877 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚680 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122391015 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145101066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚831921803948032 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267465741 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚851358167849472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267492087 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77380930 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14119069682176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933200854367744 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130112461 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121043968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚911261352277126 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122391021 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚919081784685568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19436415104512 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145074726 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚919081784685568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14122680343040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122391015 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14097935231990 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚473Ù( «£4ï/ _wandbruntime‚10421  _runtime‚10421ênÄv¦&ð—N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14121480163840 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚911410453730896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14121480163840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52740366 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 % _timestamp‚1.7884802149590864e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚911414087821756 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 _runtime‚10421.715818378 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267510732 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚919235588636160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145099293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52739704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19439747677184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145099293 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚919235562244608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130133439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚832061094337024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚832061043133952 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122411439 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14125090838016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19439819601920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122411445 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145125659 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267537104 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 _step‚474 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77393735 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122411439 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14096732872592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897313746286272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933360679474176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚851500913938944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚851500790811136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚933357042408448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267537104 C=DeviceStatsMonitor.on_train_batch_end/Ÿ}f+woversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚897313720858304÷m"'ñœN WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚919235562244608 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130133439 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚832061094337024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚832061043133952 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122411439 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14125090838016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19439819601920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122411445 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145125659 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267537104 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 _step‚474 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77393735 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122411439 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14096732872592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897313746286272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚933360679474176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚851500913938944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚851500790811136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚933357042408448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267537104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚897313720858304 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14121480163840 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚911410453730896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14121480163840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚662 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52740366 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 % _timestamp‚1.7884802149590864e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚911414087821756 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 _runtime‚10421.715818378 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267510732 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚919235588636160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145099293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52739704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19439747677184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145099293ÚS“:4ò/ _wandbruntime‚10421  _runtime‚10421.s{M]ó× 1loss/SRqqt1/detection_loss‚0.9081261400133371 1loss/SRqqt4/detection_loss‚2.4373333851496377 /loss/FBt2/detection_loss‚0.9458071936824126 %loss/total_loss‚46.93402115574302 0loss/FRt3/assignment_loss‚3.8370642942540787 epoch‚4 0loss/FRt1/assignment_loss‚1.9647331414744258 2loss/SRqqt1/assignment_loss‚1.0732001456729234 2loss/SRqqt3/assignment_loss‚1.7130620933097342 1loss/SRqqt3/detection_loss‚2.6577093575311745 .loss/FBt3/detection_loss‚2.395925833898432 2loss/SRqqt4/assignment_loss‚1.6683404754709314 /loss/FRt2/detection_loss‚1.9601301981422763 .loss/FBt4/detection_loss‚2.688895982045394 2loss/SRqqt2/assignment_loss‚1.4844398746887844 /loss/FRt4/assignment_loss‚3.445505077348036 /loss/FRt1/detection_loss‚1.7454798896484789 .loss/FRt4/detection_loss‚2.884078381692662 % _timestamp‚1.7884802149620326e+09 _runtime‚10421.717429536 0loss/FBt3/assignment_loss‚0.1300094069023651 1loss/FBt4/assignment_loss‚0.18779981867303902 /loss/FRt3/detection_loss‚3.9924367525998283 _step‚475 0loss/FRt2/assignment_loss‚2.3887805457298574 0loss/FBt1/assignment_loss‚0.2749939322238788 1loss/FBt2/assignment_loss‚0.11184125107947529 0loss/SRqqt2/detection_loss‚2.893996298313141 /loss/FBt1/detection_loss‚3.1443316861987114 trainer/global_step‚5799Al«ÍbôÜ 1loss/FBt4/assignment_loss‚0.18779981867303902 /loss/FRt3/detection_loss‚3.9924367525998283 _step‚475 0loss/FRt2/assignment_loss‚2.3887805457298574 0loss/FBt1/assignment_loss‚0.2749939322238788 1loss/FBt2/assignment_loss‚0.11184125107947529 0loss/SRqqt2/detection_loss‚2.893996298313141 /loss/FBt1/detection_loss‚3.1443316861987114 trainer/global_step‚5799 1loss/SRqqt1/detection_loss‚0.9081261400133371 1loss/SRqqt4/detection_loss‚2.4373333851496377 /loss/FBt2/detection_loss‚0.9458071936824126 %loss/total_loss‚46.93402115574302 0loss/FRt3/assignment_loss‚3.8370642942540787 epoch‚4 0loss/FRt1/assignment_loss‚1.9647331414744258 2loss/SRqqt1/assignment_loss‚1.0732001456729234 2loss/SRqqt3/assignment_loss‚1.7130620933097342 1loss/SRqqt3/detection_loss‚2.6577093575311745 .loss/FBt3/detection_loss‚2.395925833898432 2loss/SRqqt4/assignment_loss‚1.6683404754709314 /loss/FRt2/detection_loss‚1.9601301981422763 .loss/FBt4/detection_loss‚2.688895982045394 2loss/SRqqt2/assignment_loss‚1.4844398746887844 /loss/FRt4/assignment_loss‚3.445505077348036 /loss/FRt1/detection_loss‚1.7454798896484789 .loss/FRt4/detection_loss‚2.884078381692662 % _timestamp‚1.7884802149620326e+09 _runtime‚10421.717429536 0loss/FBt3/assignment_loss‚0.1300094069023651ÛžL Ÿ’õ‚(:‡ Ú•èÔÇ“ÿx gpu.0.gpu‚86.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚160.063' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0126 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A½«Œö‚(: Ú•èÔ˜¾îz% disk./.usageGB‚1161.9942092895508) disk.nvme0c0n1.out‚250649.30224609375 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚1.5625$ memory_percent‚99.96225833892822 disk.sda3.out‚ 316.90625 network.sent‚ 1.6353021e+07! disk.nvme0c0n1.in‚ 26083.03125 disk.dm-0.in‚ 26083.03125$ disk.dm-0.out‚250649.30224609375 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 3280.81640625& proc.memory.availableMB‚ 6.18359375* proc.memory.percent‚20.024514198303223 cpu‚25.210179428729166 proc.cpu.threads‚17 network.recv‚ 3.262236e+06( disk./.usagePercent‚77.9824063635035 disk.dm-1.out‚ 316.90625 disk.sda.out‚ 316.90625 disk.nvme0n1.in‚ 26083.03125' disk.nvme0n1.out‚250649.30224609375 disk.sda1.in‚0 disk.sda1.out‚0R“¨ð‘÷‚(:† é•èÔ¿íù{ gpu.0.gpu‚28.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ2ø‚(:‚ é•èԴη} cpu‚25.210246995936636 disk.dm-1.in‚ 0.00390625 network.sent‚ 1.6464e+07% disk./.usageGB‚1162.0566902160645! disk.nvme0c0n1.in‚ 26083.03125' disk.nvme0c0n1.out‚250850.982421875" disk.dm-0.out‚250850.982421875 disk.sda.out‚ 317.3046875 disk.sda3.in‚ 0.00390625 network.recv‚ 3.268864e+06) disk./.usagePercent‚77.98659951090681 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.00390625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 26083.03125% proc.memory.availableMB‚ 6.3984375 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 26083.03125 disk.drbd1260.out‚ 1.56640625% disk.nvme0n1.out‚250850.982421875* proc.memory.percent‚20.024514198303223 proc.cpu.threads‚17 disk.dm-1.out‚ 317.3046875 disk.sda3.out‚ 317.3046875$ memory_percent‚99.96094703674316# proc.memory.rssMB‚ 3280.81640625°kÁÕ‘ù‚(:† ø•èÔë–þy gpu.0.gpu‚23.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÎP‹ú‚(:€ ø•èÔâ«Úz' disk.nvme0c0n1.out‚251101.439453125 disk.sda.out‚ 317.7734375 disk.sda3.out‚ 317.7734375% disk.nvme0n1.out‚251101.439453125 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 317.7734375 disk.nvme0n1.in‚ 26083.03125 cpu‚25.2103645002036 proc.cpu.threads‚17 network.sent‚ 1.6466922e+07( disk./.usagePercent‚77.9950062863162 disk.sda1.out‚0$ memory_percent‚99.96099472045898$ proc.memory.availableMB‚6.390625# proc.memory.rssMB‚ 3280.81640625* proc.memory.percent‚20.024514198303223! disk.nvme0c0n1.in‚ 26083.03125 disk.dm-0.in‚ 26083.03125" disk.dm-0.out‚251101.439453125 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.56640625 disk.sda2.in‚0 network.recv‚ 3.272113e+06$ disk./.usageGB‚1162.181957244873Æ{k”û‚(:‰ ‡–èÔ–Ò„y' disk.nvme0n1.out‚251602.57763671875 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚99.94244575500488# proc.memory.rssMB‚ 3280.83203125 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 318.5234375 disk.drbd1260.out‚ 1.5703125 disk.nvme0n1.in‚ 26083.03125! disk.nvme0c0n1.in‚ 26083.03125$ disk.dm-0.out‚251602.57763671875* proc.memory.percent‚20.024609565734863 network.sent‚ 1.6473311e+07) disk./.usagePercent‚77.99496839720113 disk.sda1.in‚0) disk.nvme0c0n1.out‚251602.57763671875 cpu‚25.21042916129707% disk./.usageGB‚1162.1813926696777 disk.sda2.in‚0 disk.dm-0.in‚ 26083.03125 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 318.5234375% proc.memory.availableMB‚ 9.4296875 proc.cpu.threads‚17 network.recv‚ 3.276872e+06 disk.sda.out‚ 318.5234375 disk.drbd1260.in‚0‡$Nö‘ü‚(:† ‡–èÔ®²Ÿy gpu.0.gpu‚37.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚166.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.2062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡vœý‚(:‘ ––èÔöÞÉw gpu.0.gpu‚73.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.331' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.06620000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ž' þ‚(:• ––èÔ”ÚÃx disk.sda1.out‚0$ disk.nvme0c0n1.in‚26209.06640625 disk.dm-1.in‚ 0.00390625 network.sent‚ 1.6478272e+07 network.recv‚ 3.280576e+06 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 319.05078125 disk.drbd1260.in‚0' disk.nvme0n1.out‚251924.06591796875) disk.nvme0c0n1.out‚251924.06591796875 disk.sda.out‚ 319.05078125) disk./.usagePercent‚78.01017242865214 disk.drbd1260.out‚ 1.5703125$ disk.dm-0.out‚251924.06591796875$ memory_percent‚99.94091987609863% proc.memory.availableMB‚ 9.6796875* proc.memory.percent‚20.024609565734863 cpu‚25.210440408953335$ disk./.usageGB‚1162.407943725586 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚26209.06640625 disk.dm-1.out‚ 319.05078125 disk.sda.in‚ 0.00390625# proc.memory.rssMB‚ 3280.83203125 proc.cpu.threads‚17" disk.nvme0n1.in‚26209.06640625 disk.sda2.in‚0Í2û’ÿ‚(:‡ ¥–èÔ Êêw gpu.0.gpu‚66.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚240.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.1452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïï–›€ ‚(: ¥–èÔ“ÊÅx disk.sda.out‚ 319.390625 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 9.6796875 cpu‚25.210519904882478 disk.sda3.out‚ 319.390625 network.sent‚ 1.648452e+07 network.recv‚ 3.285164e+06% disk./.usageGB‚1162.4079246520996) disk.nvme0c0n1.out‚252342.97802734375 disk.dm-0.in‚26209.06640625 disk.dm-1.in‚ 0.00390625* proc.memory.percent‚20.024609565734863 disk.dm-1.out‚ 319.390625 disk.sda.in‚ 0.00390625 disk.drbd1260.out‚ 1.57421875" disk.nvme0n1.in‚26209.06640625 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3280.83203125 proc.cpu.threads‚17$ disk.dm-0.out‚252342.97802734375 disk.sda3.in‚ 0.00390625' disk.nvme0n1.out‚252342.97802734375$ memory_percent‚99.94091987609863) disk./.usagePercent‚78.01017114861446$ disk.nvme0c0n1.in‚26209.066406250ÄÇ>4 / _wandbruntime‚10514  _runtime‚10514Ç|ÁÉ‚ à ' lr-AdamW/pg2‚0.00040058606325460194 trainer/global_step‚5849 $ _timestamp‚1.788480307481707e+09 _runtime‚10514.238796231 _step‚476 ' lr-AdamW/pg1‚0.00040058606325460194©É0Ñ΃ È ' lr-AdamW/pg2‚0.00040058606325460194 trainer/global_step‚5849 $ _timestamp‚1.788480307481707e+09 _runtime‚10514.238796231 _step‚476 ' lr-AdamW/pg1‚0.00040058606325460194Ümõàj4„ / _wandbruntime‚10514  _runtime‚105146½Ë(… ŒP \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚737 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269718643 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146306428 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14239608179200 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 % _timestamp‚1.7884803074833496e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚926770689029120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78021180 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚941010270816768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14239608179200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269744989 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚941013907869184 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚941013907869184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53172764 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚918898143089434 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14214655324814 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146332768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚10514.24003201 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚926770689029120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚904679853673968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14243218840064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14218263987498 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚477 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941010270816768 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146306428 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚858488962089472 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19602763930112Aó„&܆ ‘P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚926770662637568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123412221 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131194681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53173496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123412215 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytesYAä%;$.all.peak‚ 3896742400 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚941010270816768 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146306428 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚858488962089472 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19602763930112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚737 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269718643 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146306428 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14239608179200 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 % _timestamp‚1.7884803074833496e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚926770689029120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78021180 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚941010270816768 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14239608179200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269744989 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚941013907869184 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚941013907869184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53172764 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚918898143089434 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14214655324814 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146332768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚10514.24003201 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚926770689029120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚904679853673968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14243218840064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14218263987498 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚477 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14242018660864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269763634 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919047244543204 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚926924466588160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269790006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53181607 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚926924466588160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123432639 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14217061628100 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123432639 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚926924440196608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 _runtime‚10515.366324091 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269763634 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123432645 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146330995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131215592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚858631567939072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19606005355520 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123432645 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146357361 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 trainer/global_step‚5849 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚904830182915104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269790006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14242018660864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14220670290992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53182317 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚904830208343072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146357361 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19606079377408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚941166458857472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚926924440196608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚8390255625835528„7Œ"'‹ œN UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146357361 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19606079377408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚941166458857472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚926924440196608 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚839025562583552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14242018660864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 269763634 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919047244543204 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚926924466588160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269790006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53181607 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚926924466588160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123432639 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14217061628100 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123432639 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚926924440196608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚715 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 _runtime‚10515.366324091 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269763634 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123432645 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146330995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131215592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚710 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚858631567939072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19606005355520 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123432645 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146357361 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 trainer/global_step‚5849 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚904830182915104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269790006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14242018660864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14220670290992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53182317 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚904830208343072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Þ¸Åe4Œ / _wandbruntime‚10515  _runtime‚10515”@‰¤\ Ö /loss/FBt4/detection_loss‚3.1663521006703377 %loss/total_loss‚46.05698765359961 trainer/global_step‚5849 % _timestamp‚1.7884803086117773e+09 1loss/FBt2/assignment_loss‚0.16077385474606834 /loss/FRt2/detection_loss‚1.8602160185654746 0loss/FBt4/assignment_loss‚0.2037928842846668 1loss/SRqqt1/detection_loss‚1.0127565723554302 1loss/SRqqt4/detection_loss‚2.3716817465988367 epoch‚4 1loss/SRqqt1/assignment_loss‚1.106454120041067 2loss/SRqqt4/assignment_loss‚1.2870031268008657 1loss/FBt1/assignment_loss‚0.47712637193035334 /loss/FBt1/detection_loss‚2.6758723616600038 /loss/FBt3/detection_loss‚2.3189549724260967 _step‚479 1loss/FBt3/assignment_loss‚0.18876059389367583 /loss/FBt2/detection_loss‚0.5781028904498388 .loss/FRt1/detection_loss‚1.795036359630102 0loss/FRt2/assignment_loss‚2.8136538026180675 /loss/FRt3/assignment_loss‚3.959376593430837 /loss/FRt4/assignment_loss‚3.027698735396067 1loss/SRqqt3/detection_loss‚2.3580539439405714 0loss/FRt1/assignment_loss‚1.9944812747148366 1loss/SRqqt2/assignment_loss‚1.614923525940288 2loss/SRqqt3/assignment_loss‚1.6995088580463613 0loss/SRqqt2/detection_loss‚3.062392619523135 _runtime‚10515.368119526 .loss/FRt3/detection_loss‚2.978259806831678 /loss/FRt4/detection_loss‚3.3457545191049576¯®§[Ž Û %loss/total_loss‚46.05698765359961 trainer/global_step‚5849 % _timestamp‚1.7884803086117773e+09 1loss/FBt2/assignment_loss‚0.16077385474606834 /loss/FRt2/detection_loss‚1.8602160185654746 0loss/FBt4/assignment_loss‚0.2037928842846668 1loss/SRqqt1/detection_loss‚1.0127565723554302 1loss/SRqqt4/detection_loss‚2.3716817465988367 epoch‚4 1loss/SRqqt1/assignment_loss‚1.106454120041067 2loss/SRqqt4/assignment_loss‚1.2870031268008657 1loss/FBt1/assignment_loss‚0.47712637193035334 /loss/FBt1/detection_loss‚2.6758723616600038 /loss/FBt3/detecti´,Êon_loss‚2.3189549724260967 _step‚479 1loss/FBt3/assignment_loss‚0.18876059389367583 /loss/FBt2/detection_loss‚0.5781028904498388 .loss/FRt1/detection_loss‚1.795036359630102 0loss/FRt2/assignment_loss‚2.8136538026180675 /loss/FRt3/assignment_loss‚3.959376593430837 /loss/FRt4/assignment_loss‚3.027698735396067 1loss/SRqqt3/detection_loss‚2.3580539439405714 0loss/FRt1/assignment_loss‚1.9944812747148366 1loss/SRqqt2/assignment_loss‚1.614923525940288 2loss/SRqqt3/assignment_loss‚1.6995088580463613 0loss/SRqqt2/detection_loss‚3.062392619523135 _runtime‚10515.368119526 .loss/FRt3/detection_loss‚2.978259806831678 /loss/FRt4/detection_loss‚3.3457545191049576 /loss/FBt4/detection_loss‚3.1663521006703377ßÍጜ ‚(:‘ ÖèÔÄ´¤z$ memory_percent‚99.93877410888672$ proc.memory.availableMB‚10.03125" disk.nvme0n1.in‚26209.06640625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚26209.06640625# proc.memory.rssMB‚ 3280.87109375* proc.memory.percent‚20.024847984313965 network.sent‚ 1.6587219e+07) disk./.usagePercent‚78.02541409321067 disk.sda2.in‚0 disk.dm-0.in‚26209.06640625# disk.dm-0.out‚253107.4052734375 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 320.51953125 disk.drbd1260.in‚0% disk./.usageGB‚1162.6350555419922( disk.nvme0c0n1.out‚253107.4052734375 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 320.51953125 disk.drbd1260.out‚1.578125 cpu‚25.210669951735095 proc.cpu.threads‚17 network.recv‚ 3.294862e+06& disk.nvme0n1.out‚253107.4052734375 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 320.51953125 disk.sda.in‚ 0.00390625<Þ”I ‚(:… ÖèÔäí£{ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yøó4‘ ‚(:… Ò–èÔ÷¥»y$ disk./.usageGB‚1162.634017944336 disk.drbd1260.out‚1.578125# disk.nvme0c0n1.in‚ 26374.5078125 disk.sda.in‚ 0.00390625 disk.sda.out‚ 320.96875 disk.sda3.out‚ 320.96875# proc.memory.rssMB‚ 3280.87109375* proc.memory.percent‚20.024847984313965 network.sent‚ 1.6684089e+07) disk./.usagePercent‚78.02534445916135! disk.nvme0n1.in‚ 26374.5078125 disk.sda2.in‚0( disk.nvme0c0n1.out‚253262.8427734375 disk.dm-0.in‚ 26374.5078125 disk.sda3.in‚ 0.00390625# memory_percent‚99.9387264251709 disk.drbd1260.in‚0& disk.nvme0n1.out‚253262.8427734375 disk.sda2.out‚0 disk.sda1.in‚0# disk.dm-0.out‚253262.8427734375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 320.96875& proc.memory.availableMB‚ 10.0390625 network.recv‚ 3.299523e+06 disk.sda1.out‚0 cpu‚25.210752864337113 proc.cpu.threads‚17›i—C’ ‚(:’ Ò–èÔÑÈçy gpu.0.gpu‚53.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚210.798' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.159600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Ȇ)‘“ ‚(:† á–èÔ‚È°y gpu.0.gpu‚29.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚116.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xçö‰” ‚(:þ á–èÔó¸ºy network.sent‚ 1.6690491e+07 disk.dm-0.out‚ 253642.625 disk.dm-1.out‚ 321.25390625 disk.sda.out‚ 321.25390625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.93875026702881) disk./.usagePercent‚78.02534394714628% disk./.usageGB‚1162.6340103149414# disk.nvme0c0n1.in‚ 26374.5078125 disk.dm-0.in‚ 26374.5078125 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 26374.5078125 disk.nvme0n1.out‚ 253642.625 disk.sda2.out‚0 network.recv‚ 3.303996e+06! disk.nvme0c0n1.out‚ 253642.625 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 321.25390625 disk.drbd1260.out‚ 1.58203125' proc.memory.availableMB‚ 10.03515625# proc.memory.rssMB‚ 3280.87109375* proc.memory.percent‚20.024847984313965 cpu‚25.210808203118972 proc.cpu.threads‚17\ηüž• ‚(:“ ð–èÔجÓy disk.sda3.out‚ 321.6171875 network.recv‚ 3.306287e+06) disk./.usagePercent‚78.02377410894633 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 26374.5078125) disk.nvme0c0n1.out‚254193.68115234375 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.58203125 disk.sda.in‚ 0.00390625 disk.sda.out‚ 321.6171875 disk.sda3.in‚ 0.00390625' proc.memory.availableMB‚ 10.03515625* proc.memory.percent‚20.024847984313965 cpu‚25.210885735618422 proc.cpu.threads‚17 network.sent‚ 1.6694337e+07 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚99.93875026702881# proc.memory.rssMB‚ 3280.87109375% disk./.usageGB‚1162.6106185913086! disk.nvme0n1.in‚ 26374.5078125' disk.nvme0n1.out‚254193.68115234375 disk.sda2.in‚0 disk.dm-0.in‚ 26374.5078125$ disk.dm-0.out‚254193.68115234375 disk.dm-1.out‚ 321.6171875‰-cà– ‚(:„ ð–èÔ“ÌŠz gpu.0.gpu‚28.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚39.0 gpu.0.powerWatts‚180.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{¥Éœ— ‚(:‘ ÿ–èÔŠÄ×y gpu.0.gpu‚64.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚258.96' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.791999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:g÷p˜ ‚(:’ ÿ–èÔ“†›z) disk.nvme0c0n1.out‚254527.60986328125 disk.dm-0.in‚ 26374.5078125 disk.dm-1.in‚ 0.00390625$ memory_percent‚99.93875026702881' disk.nvme0n1.out‚254527.60986328125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 26374.5078125 disk.sda.in‚ 0.00390625* proc.memory.percent‚20.024847984313965 cpu‚25.210914582388238 proc.cpu.threads‚17 network.recv‚ 3.310954e+06! disk.nvme0n1.in‚ 26374.5078125$ disk.dm-0.out‚254527.60986328125' proc.memory.availableMB‚ 10.03515625# proc.memory.rssMB‚ 3280.87109375) disk./.usagePercent‚78.02534522718395% disk./.usageGB‚1162.6340293884277 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 322.1484375 disk.sda.out‚ 322.1484375 network.sent‚ 1.6700742e+07 disk.sda3.out‚ 322.1484375 disk.drbd1260.out‚ 1.5859375 disk.sda2.out‚0‰‘˜æœ™ ‚(:‘ Ž—èÔŸ±Ìz gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.878' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.97559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D?£T£š ‚(:˜ Ž—èÔ¸Æú{% disk./.usageGB‚1162.6340293884277$ disk.nvme0c0n1.in‚26518.01171875) disk.nvme0c0n1.out‚254874.49755859375 disk.dm-1.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚26518.01171875 disk.dm-1.out‚ 322.54296875 disk.sda.out‚ 322.54296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.5859375$ memory_percent‚99.93875026702881' proc.memory.availableMB‚ 10.03515625* proc.memory.percent‚20.024847984313965 network.recv‚ 3.314264e+06$ disk.dm-0.out‚254874.49755859375 disk.sda3.in‚ 0.00390625# proc.memory.rssMB‚ 3280.87109375 cpu‚25.210937299065456 proc.cpu.threads‚17 network.sent‚ 1.6705652e+07 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 322.54296875" disk.nvme0n1.in‚26518.01171875' disk.nvme0n1.out‚254874.49755859375 disk.sda1.out‚0) disk./.usagePercent‚78.02534522718395œÆ04› / _wandbruntime‚10605  _runtime‚106057P-=Êœ Ä ' lr-AdamW/pg1‚0.00039599843858936204 ' lr-AdamW/pg2‚0.00039599843858936204 trainer/global_step‚5899 % _timestamp‚1.7884803985105464e+09 _runtime‚10605.266302594 _step‚480•ÆþyÏ É trainer/global_step‚5899 % _timestamp‚1.7884803985105464e+09 _runtime‚10605.266302594 _step‚480 ' lr-AdamW/pg1‚0.00039599843858936204 ' lr-AdamW/pg2‚0.00039599843858936204àÁ÷E4ž / _wandbruntime‚10605  _runtime‚10605½hN:(Ÿ ŒP M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934459540589568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147564470 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147538130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271997891 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚865619866951168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124433421 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚912196341158736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚10605.267762716 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 _step‚481 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚948823324318208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132276407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14334984080322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78661435 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚845850842847232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19769075307008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5899 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934459566981120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14360146676224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78661430 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124433415 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚845850791644160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚948819687265792 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271971545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19769149342208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚948819687265792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚912196315730768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚948823324318208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53615692 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14338592743006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14363757337088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14363757337088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124433415 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147564470 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚845850842847232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19769075307008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5899 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934459566981120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14360146676224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78661430 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124433415 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚845850791644160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚948819687265792 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271971545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19769149342208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚948819687265792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚912196315730768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚948823324318208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53615692 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14338592743006 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14363757337088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14363757337088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124433415 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147564470 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147564470 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147538130 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271997891 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚865619866951168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124433421 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚912196341158736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 _runtime‚10605.267762716 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 _step‚481 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚948823324318208ázƨ4¡ / _wandbruntime‚10607  _runtime‚10607ß@[ñ ¢ —N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124453839 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124453845 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934613318148608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53623845 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272042908 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272016536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934613344540160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124453839 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14337390383608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚948975875306496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚865762504653312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272042908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚912346670399872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚912346644971904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚948979512372224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚948979512372224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272016536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132298080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124453845 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147562697 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147589063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53624551 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚845990082033152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934613318148608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14366167832064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132298791 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19772496642048 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14362557157888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147589063 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14366167832064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 % _timestamp‚1.7884804003546202e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147562697 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78674240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424ÑÆD_"'£ œN ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934613318148608 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14366167832064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132298791 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19772496642048 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚5899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14362557157888 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147589063 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14366167832064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 % _timestamp‚1.7884804003546202e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147562697 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78674240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124453839 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124453845 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚934613318148608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53623845 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272042908 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272016536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934613344540160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124453839 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14337390383608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚948975875306496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚865762504653312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272042908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚912346670399872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚912346644971904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚948979512372224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚948979512372224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272016536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132298080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124453845 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147562697 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147589063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53624551 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚845990082033152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187â'÷Ù4¤ / _wandbruntime‚10607  _runtime‚10607!"ÃX¥ Ò /loss/FRt1/detection_loss‚1.5953833536528397 /loss/FRt3/assignment_loss‚3.878895806229633 /loss/FRt4/assignment_loss‚3.968538820743561 1loss/SRqqt2/assignment_loss‚1.742965938953253 2loss/SRqqt3/assignment_loss‚1.8244211906972139 /loss/FRt4/detection_loss‚3.2642645835876465 _runtime‚10607.111962705 0loss/FRt1/assignment_loss‚2.0368164109217153 2loss/SRqqt1/assignment_loss‚1.0989393941870256 /loss/FBt2/detection_loss‚0.6316823618220432 2loss/SRqqt4/assignment_loss‚1.4591714484849945 0loss/FBt3/assignment_loss‚0.3533793512712388 0loss/SRqqt1/detection_loss‚1.094997209817672 1loss/FBt4/assignment_loss‚0.31712163694246553 .loss/FRt3/detection_loss‚2.918693809405617 /loss/FRt2/assignment_loss‚3.036266576825527 0loss/FBt1/assignment_loss‚0.5684115527765243 &loss/total_loss‚48.105905456804464 % _timestamp‚1.7884804003564875e+09 .loss/FRt2/detection_loss‚2.031349106217947 1loss/SRqqt4/detection_loss‚2.2548109935596585 .loss/FBt3/detection_loss‚2.334871566295624 /loss/FBt4/detection_loss‚3.1433323860168456 0loss/FBt2/assignment_loss‚0.2330497681028838 1loss/SRqqt2/detection_loss‚3.1354115834602942 0loss/SRqqt3/detection_loss‚2.771898541761481 .loss/FBt1/detection_loss‚2.411232065070759 epoch‚4 trainer/global_step‚5899 _step‚483{*v]¦ × 1loss/FBt4/assignment_loss‚0.31712163694246553 .loss/FRt3/detection_loss‚2.918693809405617 /loss/FRt2/assignment_loss‚3.036266576825527 0loss/FBt1/assignment_loss‚0.5684115527765243 &loss/total_loss‚48.105905456804464 % _timestamp‚1.7884804003564875e+09 .loss/FRt2/detection_loss‚2.031349106217947 1loss/SRqqt4/detection_loss‚2.2548109935596585 .loss/FBt3/detection_loss‚2.334871566295624 /loss/FBt4/detection_loss‚3.1433323860168456 0loss/FBt2/assignment_loss‚0.2330497681028838 1loss/SRqqt2/detection_loss‚3.1354115834602942 0loss/SRqqt3/detection_loss‚2.771898541761481 .loss/FBt1/detection_loss‚2.411232065070759 epoch‚4 trainer/global_step‚5899 _step‚483 /loss/FRt1/detection_loss‚1.5953833536528397 /loss/FRt3/assignment_loss‚3.878895806229633 /loss/FRt4/assignment_loss‚3.968538820743561 1loss/SRqqt2/assignment_loss‚1.742965938953253 2loss/SRqqt3/assignment_loss‚1.8244211906972139 /loss/FRt4/detection_loss‚3.2642645835876465 _runtime‚10607.111962705 0loss/FRt1/assignment_loss‚2.0368164109217153 2loss/SRqqt1/assignment_loss‚1.0989393941870256 /loss/FBt2/detection_loss‚0.6316823618220432 2loss/SRqqt4/assignment_loss‚1.4591714484849945 0loss/FBt3/assignment_loss‚0.3533793512712388 0loss/SRqqt1/detection_loss‚1.094997209817672ãKö¬Gœ§ ‚(:‘ —èÔºº˜z gpu.0.gpu‚63.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.14' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.827999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ÑN¬¨ ‚(:„ —èÔ…²½z cpu‚25.21102265080351" disk.nvme0n1.in‚26518.01171875' disk.nvme0n1.out‚255245.58349609375 disk.sda1.in‚0 disk.dm-1.out‚322.875 disk.sda.in‚ 0.00390625$ memory_percent‚99.94349479675293% proc.memory.availableMB‚ 9.2578125 network.recv‚ 3.319215e+06 disk.sda2.in‚0 disk.dm-0.in‚26518.01171875 disk.dm-1.in‚ 0.00390625 disk.sda.out‚322.875 disk.sda3.in‚ 0.00390625# proc.memory.rssMB‚ 3280.87109375 network.sent‚ 1.671192e+07) disk./.usagePercent‚78.02536468375656$ disk.nvme0c0n1.in‚26518.01171875 disk.sda3.out‚322.875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.58984375* proc.memory.percent‚20.024847984313965 proc.cpu.threads‚17# disk./.usageGB‚1162.63431930542 disk.sda2.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚255245.58349609375$ disk.dm-0.out‚255245.58349609375ëØ †Ÿ© ‚(:” ¬—èÔÒžy* proc.memory.percent‚20.024847984313965 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.sda.out‚ 323.2734375 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 323.2734375 network.sent‚ 1.6819484e+07 network.recv‚ 3.32412e+06% disk./.usageGB‚1162.6343383789062) disk.nvme0c0n1.out‚255398.95849609375 disk.dm-0.in‚26518.01171875$ disk.dm-0.out‚255398.95849609375 disk.dm-1.out‚ 323.2734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.58984375' disk.nvme0n1.out‚255398.95849609375 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.02536596379423$ disk.nvme0c0n1.in‚26518.01171875 disk.dm-1.in‚ 0.00390625$ memory_percent‚99.94280338287354# proc.memory.rssMB‚ 3280.87109375 cpu‚25.211060025922386" disk.nvme0n1.in‚26518.01171875 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 9.371093755)V‘ª ‚(:† ¬—èÔƒµÀz gpu.0.gpu‚53.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚205.955' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.191 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÕSü« ‚(:’ »—èÔÜó‚x' disk.nvme0n1.out‚255938.30517578125 disk.sda2.in‚0$ disk.nvme0c0n1.in‚26518.01171875 disk.dm-0.in‚26518.01171875 disk.sda.out‚ 323.77734375 disk.sda3.out‚ 323.77734375 disk.drbd1260.out‚1.59375$ memory_percent‚99.94301795959473 network.recv‚ 3.328966e+06) disk./.usagePercent‚78.02536468375656" disk.nvme0n1.in‚26518.01171875 disk.sda1.in‚0 disk.dm-1.out‚ 323.77734375# proc.memory.rssMB‚ 3280.87109375* proc.memory.percent‚20.024847984313965 cpu‚25.211189927965556 disk.sda2.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚255938.30517578125 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚ 1.6825844e+07# disk./.usageGB‚1162.63431930542$ disk.dm-0.out‚255938.30517578125% proc.memory.availableMB‚ 9.3359375MK÷W¬ ‚(:… »—èÔòø²x gpu.0.gpu‚3.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gA­ ‚(:’ Ê—èÔÊÁÇx network.sent‚ 1.6830822e+07) disk./.usagePercent‚78.02536698782436$ disk.dm-0.out‚256259.90771484375" disk.nvme0n1.in‚26655.37109375 disk.sda2.out‚0$ memory_percent‚99.94313716888428* proc.memory.percent‚20.024943351745605 cpu‚25.211311107953644 disk.dm-0.in‚26655.37109375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 324.2265625 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 324.2265625' disk.nvme0n1.out‚256259.90771484375 disk.sda2.in‚0 disk.sda1.in‚0% disk./.usageGB‚1162.6343536376953$ disk.nvme0c0n1.in‚26655.37109375) disk.nvme0c0n1.out‚256259.90771484375 disk.drbd1260.in‚0 disk.drbd1260.out‚1.59375 disk.sda1.out‚0& proc.memory.availableMB‚ 9.31640625# proc.memory.rssMB‚ 3280.88671875 network.recv‚ 3.332346e+06 disk.sda.out‚ 324.2265625 disk.sda3.in‚ 0.00390625 proc.cpu.threads‚17¤•Ê’® ‚(:‡ Ê—èÔ»ºñy gpu.0.gpu‚51.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚192.826' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.5652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09¾h#¡¯ ‚(:– Ù—èÔظÇy network.recv‚ 3.33681e+06 disk.sda3.out‚ 324.50390625 disk.drbd1260.out‚ 1.59765625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚26655.37109375$ memory_percent‚99.94475841522217& proc.memory.availableMB‚ 9.05078125) disk./.usagePercent‚78.02536570778669 disk.sda.in‚ 0.00390625 disk.sda.out‚ 324.50390625 disk.sda3.in‚ 0.00390625' disk.nvme0n1.out‚256760.79638671875$ disk.dm-0.out‚256760.79638671875* proc.memory.percent‚20.025038719177246 proc.cpu.threads‚17$ disk./.usageGB‚1162.634334564209 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0" disk.nvme0n1.in‚26655.37109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 cpu‚25.211429738703178 disk.dm-1.out‚ 324.50390625) disk.nvme0c0n1.out‚256760.79638671875 disk.dm-0.in‚26655.37109375# proc.memory.rssMB‚ 3280.90234375 network.sent‚ 1.6836868e+07á¯+° ‚(:… Ù—èÔþ°Òy gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ôaˆ¢± ‚(:— è—èÔØû‚y" disk.nvme0n1.in‚26655.37109375 disk.sda2.in‚0$ disk.nvme0c0n1.in‚26655.37109375) disk.nvme0c0n1.out‚257203.52685546875 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 network.recv‚ 3.340176e+06' disk.nvme0n1.out‚257203.52685546875 disk.sda2.out‚0 disk.sda1.in‚0$ disk.dm-0.out‚257203.52685546875 disk.sda.out‚ 324.88671875 disk.drbd1260.out‚ 1.59765625# proc.memory.rssMB‚ 3280.90234375) disk./.usagePercent‚78.02538106823874 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 324.88671875 disk.drbd1260.in‚0$ memory_percent‚99.94475841522217* proc.memory.percent‚20.025038719177246 disk.sda1.out‚0 disk.dm-0.in‚26655.37109375 disk.dm-1.out‚ 324.88671875& proc.memory.availableMB‚ 9.05078125 cpu‚25.211532213972284 proc.cpu.threads‚17 network.sent‚ 1.6841888e+07$ disk./.usageGB‚1162.634563446045§Ìn,² ‚(:’ è—èÔÇÞ–y gpu.0.gpu‚52.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚204.657' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.931400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þµbº4³ / _wandbruntime‚10700  _runtime‚107006|ٶȴ  & lr-AdamW/pg2‚0.0003913962109432919 trainer/global_step‚5949 % _timestamp‚1.7884804934277487e+09 _runtime‚10700.183493222 _step‚484 & lr-AdamW/pg1‚0.0003913962109432919 ”þ͵ Ç & lr-AdamW/pg1‚0.0003913962109432919 & lr-AdamW/pg2‚0.0003913962109432919 trainer/global_step‚5949 % _timestamp‚1.7884804934277487e+09 _runtime‚10700.183493222 _step‚484䣄Eî4¶ / _wandbruntime‚10700  _runtime‚10700~.–f · ŒP GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14484295834112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚956629103714816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125454621 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚934171724714050 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _step‚485 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 H>DeviceStatsMonitor.on_train_batch_start/ac‡óéÝtive.small_pool.peak‚32002 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚956632740767232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79301685 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148796172 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚942148418541568 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚919712803215536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 trainer/global_step‚5949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79301680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚872750490221056 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚919712777787568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚942148444933120 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚956629103714816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125454615 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚934168090623398 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14455312835830 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚852815311093760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148796172 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19935253162496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788480493428943e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14480685173248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14480685173248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125454621 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚872750615459328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133358945 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚934168090623398 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14455312835830 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚709 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚852815311093760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148796172 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19935253162496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788480493428943e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14480685173248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14480685173248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125454621 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚872750615459328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133358945 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14484295834112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚956629103714816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125454621 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚714 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125238272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚934171724714050 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 _step‚485 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚956632740767232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79301685 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148796172 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚942148418541568 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚919712803215536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 trainer/global_step‚5949åLdñú4¹ / _wandbruntime‚10701  _runtime‚10701]†û'º —N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14486706329088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942302196100608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148820765 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚852954652685824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884804946867182e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274295810 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125475039 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚919863132456672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133380523 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54066744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚852954601482752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942302222492160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148794399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 _runtime‚10701.445977954 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125475045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79314490 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚942302196100608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79314485 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14457719139116 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19938544325632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54066038 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚934324460258680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148820765 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125475039 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125475045 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 trainer/global_step‚5949 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚956788928821248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133381234 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14461327802008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚956785291755520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19938470303744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14483095654912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14486706329088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274295810 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚956788928821248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14483095654912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274269438 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176˶n» œN C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7884804946867182e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274295810 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125475039 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚919863132456672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133380523 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54066744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚852954601482752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942302222492160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148794399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 _runtime‚10701.445977954 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125475045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79314490 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚942302196100608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79314485 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14457719139116 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19938544325632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54066038 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚934324460258680 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148820765 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125475039 HBDeviceStatsMonitor.on_train_batch_end/f;Í´oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125475045 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 trainer/global_step‚5949 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚956788928821248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133381234 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14461327802008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚956785291755520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19938470303744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14483095654912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14486706329088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274295810 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚956788928821248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14483095654912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274269438 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14486706329088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942302196100608 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148820765 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚852954652685824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0æUÖ®4¼ / _wandbruntime‚10701  _runtime‚10701fZWlU½ Ï _runtime‚10701.450223598 trainer/global_step‚5949 -loss/FRt4/detection_loss‚3.19177628159523 /loss/SRqqt3/detection_loss‚2.37839248224541 2loss/SRqqt2/assignment_loss‚1.2531971380114555 2loss/SRqqt4/assignment_loss‚1.9220855962939378 _step‚487 0loss/FBt4/assignment_loss‚0.3139351178787448 /loss/FRt1/detection_loss‚1.7271089831318485 0loss/FRt1/assignment_loss‚2.0767716124876223 0loss/FBt3/assignment_loss‚0.2744344118634672 /loss/FBt3/detection_loss‚2.6872684028413563 /loss/FRt2/detection_loss‚1.7809561040189306 0loss/SRqqt1/assignment_loss‚1.28010973954437 1loss/SRqqt3/assignment_loss‚2.140394175494159 0loss/FBt1/assignment_loss‚0.5446784508725008 1loss/FBt2/assignment_loss‚0.17012874142470757 0loss/SRqqt2/detection_loss‚2.658208879828453 -loss/FRt3/detection_loss‚3.38802616704594 1loss/SRqqt4/detection_loss‚2.0735207926936265 /loss/FRt3/assignment_loss‚4.420745551586151 0loss/FRt4/assignment_loss‚3.1135965526103973 /loss/FBt1/detection_loss‚2.4142634918292365 %loss/total_loss‚46.77616072106038 % _timestamp‚1.7884804946937504e+09 0loss/FRt2/assignment_loss‚2.4963480123018815 1loss/SRqqt1/detection_loss‚1.0104940408855918 /loss/FBt2/detection_loss‚0.7883923982143781 .loss/FBt4/detection_loss‚2.671327596360987 epoch‚4®Z^ŒZ¾ Ô 0loss/FBt4/assignment_loss‚0.3139351178787448 /loss/FRt1/detection_loss‚1.7271089831318485 0loss/FRt1/assignment_loss‚2.0767716124876223 0loss/FBt3/assignment_loss‚0.2744344118634672 /loss/FBt3/detection_loss‚2.6872684028413563 /loss/FRt2/detection_loss‚1.7809561040189306 0loss/SRqqt1/assignment_loss‚1.28010973954437 1loss/SRqqt3/assignment_loss‚2.140394175494159 0loss/FBt1/assignment_loss‚0.5446784508725008 1loss/FBt2/assignment_loss‚0.17012874142470757 0loss/SRqqt2/detection_loss‚2.658208879828453 -loss/FRt3/detection_loss‚3.38802616704594 1loss/SRqqt4/detection_loss‚2.0735207926936265 /loss/FRt3/assignment_loss‚4.420745551586151 0loss/FRt4/assignment_loss‚3.1135965526103973 /loss/FBt1/detection_loss‚2.4142634918292365 %loss/total_loss‚46.77616072106038 % _timestamp‚1.7884804946937504e+09 0loss/FRt2/assignment_loss‚2.4963480123018815 1loss/SRqqt1/detection_loss‚1.0104940408855918 /loss/FBt2/detection_loss‚0.7883923982143781 .loss/FBt4/detection_loss‚2.671327596360987 epoch‚4 _runtime‚10701.450223598 trainer/global_step‚5949 -loss/FRt4/detection_loss‚3.19177628159523 /loss/SRqqt3/detection_loss‚2.37839248224541 2loss/SRqqt2/assignment_loss‚1.2531971380114555 2loss/SRqqt4/assignment_loss‚1.9220855962939378 _step‚487çHÜä¿ ‚(:’ ÷—èÔÕðv gpu.0.gpu‚66.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.004' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.400800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä1ÚM˜À ‚(: ÷—èÔî¯Âx disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 325.3984375 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 325.3984375 network.sent‚ 1.6848144e+07 disk.dm-0.in‚26655.37109375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚98.83837699890137" proc.memory.rssMB‚ 3091.6953125% disk./.usageGB‚1162.6346244812012 disk.drbd1260.out‚ 1.6015625 disk.sda2.out‚0' proc.memory.availableMB‚ 190.3203125* proc.memory.percent‚18.870210647583008 cpu‚25.21156497432284 proc.cpu.threads‚17) disk./.usagePercent‚78.02538516435929' disk.nvme0c0n1.out‚257381.279296875" disk.dm-0.out‚257381.279296875 disk.dm-1.out‚ 325.3984375 disk.sda.in‚ 0.00390625" disk.nvme0n1.in‚26655.37109375% disk.nvme0n1.out‚257381.279296875 network.recv‚ 3.345124e+06$ disk.nvme0c0n1.in‚26655.37109375EõzB•Á ‚(:Š †˜èÔäø„x proc.cpu.threads‚17 network.sent‚ 1.695951e+07) disk./.usagePercent‚78.02538388432161" disk.nvme0n1.in‚26778.63671875' disk.nvme0c0n1.out‚257536.677734375 disk.dm-0.in‚26778.63671875 disk.sda.out‚ 325.765625 disk.sda3.in‚ 0.00390625 network.recv‚ 3.357349e+06% disk.nvme0n1.out‚257536.677734375 disk.sda2.in‚0" disk.dm-0.out‚257536.677734375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6015625" proc.memory.rssMB‚ 3096.2265625 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚26778.63671875 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 325.765625 disk.sda.in‚ 0.00390625) proc.memory.percent‚18.89786720275879% disk./.usageGB‚1162.6346054077148 disk.sda3.out‚ 325.765625$ memory_percent‚98.86467456817627( proc.memory.availableMB‚ 186.01171875 cpu‚25.211643679970244bzÙ¾œ ‚(:‘ †˜èÔáëïx gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.558' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.31159999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽEÆd¢à ‚(:— •˜èÔ˜¸y disk.sda3.out‚ 326.14453125 disk.drbd1260.out‚ 1.60546875' disk.nvme0n1.out‚257834.63427734375 disk.sda3.in‚ 0.00390625$ memory_percent‚98.89109134674072( proc.memory.availableMB‚ 181.68359375! proc.memory.rssMB‚ 3100.734375) proc.memory.percent‚18.92538070678711 proc.cpu.threads‚17 network.sent‚ 1.6965718e+07% disk./.usageGB‚1162.6345481872559 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0 cpu‚25.211713138279087" disk.nvme0n1.in‚26778.63671875 disk.sda2.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚26778.63671875 disk.dm-0.in‚26778.63671875$ disk.dm-0.out‚257834.63427734375 disk.dm-1.out‚ 326.14453125 disk.sda.in‚ 0.00390625 network.recv‚ 3.361849e+06) disk./.usagePercent‚78.02538004420862 disk.sda2.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚257834.63427734375 disk.sda.out‚ 326.14453125¶Jš’Ä ‚(:‡ •˜èÔûæÊy gpu.0.gpu‚92.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚43.0 gpu.0.powerWatts‚253.559' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wt“œÅ ‚(:‘ ¤˜èÔ—½èy gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚242.078' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.415600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘èW1›Æ ‚(: ¤˜èÔ£»Óz disk.sda.out‚ 326.8046875 disk.drbd1260.out‚ 1.60546875$ memory_percent‚98.91576766967773( proc.memory.percent‚18.9547061920166 network.sent‚ 1.6970596e+07) disk./.usagePercent‚78.02538158025382 disk.sda1.in‚0 disk.dm-0.in‚26778.63671875 disk.sda.in‚ 0.00390625 cpu‚25.2117831884473 disk.sda1.out‚0$ disk.nvme0c0n1.in‚26778.63671875) disk.nvme0c0n1.out‚258253.31396484375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 326.8046875 disk.sda3.out‚ 326.8046875 disk.drbd1260.in‚0& proc.memory.availableMB‚ 177.640625 network.recv‚ 3.365038e+06% disk./.usageGB‚1162.6345710754395' disk.nvme0n1.out‚258253.31396484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.00390625" proc.memory.rssMB‚ 3105.5390625 proc.cpu.threads‚17" disk.nvme0n1.in‚26778.63671875$ disk.dm-0.out‚258253.31396484375³’Ç ‚(:‡ ³˜èÔÝä¼y gpu.0.gpu‚78.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚253.857' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7714 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pþ–ˆÈ ‚(:… ³˜èÔŠ¹õz disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚26778.63671875( disk./.usagePercent‚78.0237881893607 disk.sda.in‚ 0.00390625 disk.sda.out‚ 327.5859375 disk.sda3.out‚ 327.5859375% disk.nvme0c0n1.out‚258684.2734375 disk.dm-0.in‚26778.63671875 disk.dm-0.out‚258684.2734375 cpu‚25.211765657789982 network.sent‚ 1.6977092e+07" disk.nvme0n1.in‚26778.63671875 disk.dm-1.in‚ 0.00390625# memory_percent‚98.9513635635376( proc.memory.availableMB‚ 171.80859375! proc.memory.rssMB‚ 3111.171875* proc.memory.percent‚18.989086151123047% disk./.usageGB‚1162.6108283996582 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 327.5859375 proc.cpu.threads‚17 network.recv‚ 3.370329e+06 disk.sda3.in‚ 0.00390625 disk.drbd1260.out‚1.609375# disk.nvme0n1.out‚258684.2734375ü°4É / _wandbruntime‚10782  _runtime‚10782©~NÆÊÊ Ä _runtime‚10782.739021467 _step‚488 ' lr-AdamW/pg1‚0.00038678033941373484 ' lr-AdamW/pg2‚0.00038678033941373484 trainer/global_step‚5999 % _timestamp‚1.7884805759834058e+09ÅéDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126475815 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚941804881435706 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150001534 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚964438520163840 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚964438520163840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14601223670272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 _runtime‚10782.74050301 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚927229239844368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134440675 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79941930 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150001534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚859779881746432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14601223670272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚949837322885120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14604834331136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20101123035136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949837296493568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126475821 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126475821 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚949837296493568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 % _timestamp‚1.7884805759847808e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134441412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54499477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20101199167488׬Wo×!Î ‘P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚949837322885120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79941935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54498745 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276477349 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150027874 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126475815 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚941804881435706 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150001534 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚964438520163840 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚964438520163840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14601223670272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 _runtime‚10782.74050301 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚927229239844368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134440675 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79941930 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150001534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚859779881746432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14601223670272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚949837322885120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14604834331136 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20101123035136 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚949837296493568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126475821 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126475821 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚949837296493568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 % _timestamp‚1.7884805759847808e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134441412 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54499477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 127335424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20101199167488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134462318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79954735 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚859919120932352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚949991074052608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14607244826112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚964598345270272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276548712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150026101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54508284 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20104522988032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚927379569085504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚964598345270272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚949991100444160 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134463024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14603634151936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14581656557516 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 % _timestamp‚1.7884805770839367e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136̤"'Ñ œN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 trainer/global_step‚5999 _runtime‚10783.840732334 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276548712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚964594708204544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚859919120932352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚949991074052608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 127322112 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14607244826112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚964598345270272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276548712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150026101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54508284 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20104522988032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚927379569085504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚964598345270272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚949991100444160 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134463024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14603634151936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14581656557516 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 % _timestamp‚1.7884805770839367e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚880023567801344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20104446868992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14607244826112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126496245 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150052467 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150052467 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134462318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79954735 P²Ø’Õ ‚(:‡ ˜èÔâ¦ùw gpu.0.gpu‚76.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚117.539' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º´M¥„Ö ‚(:ù ˜èÔÓÉÂ{( proc.memory.availableMB‚ 164.16015625# proc.memory.rssMB‚ 3118.58203125# disk.nvme0n1.out‚259115.3203125 disk.sda1.out‚0% disk.nvme0c0n1.out‚259115.3203125 disk.dm-1.out‚ 328.0859375 disk.sda.in‚ 0.00390625 disk.drbd1260.out‚1.609375) disk./.usagePercent‚77.97410301513796 disk.nvme0n1.in‚ 26793.375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.out‚ 328.0859375 disk.drbd1260.in‚0$ memory_percent‚98.99804592132568 network.sent‚ 1.6982082e+07 disk.dm-0.in‚ 26793.375 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 328.0859375* proc.memory.percent‚19.034314155578613 cpu‚25.21158561701696 proc.cpu.threads‚17 disk.dm-0.out‚259115.3203125 network.recv‚ 3.373851e+06% disk./.usageGB‚1161.8704833984375 disk.nvme0c0n1.in‚ 26793.375 disk.sda3.in‚ 0.00390625`L ‡ž× ‚(:“ јèÔ¡Áîw disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 26914.2578125) disk.nvme0c0n1.out‚259500.34912109375 disk.dm-0.in‚ 26914.2578125 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 328.69921875 disk.drbd1260.out‚ 1.61328125( proc.memory.availableMB‚ 157.01953125 cpu‚25.211426678346445 proc.cpu.threads‚17) disk./.usagePercent‚77.97410147909275$ disk./.usageGB‚1161.870460510254$ disk.dm-0.out‚259500.34912109375 disk.dm-1.in‚ 0.00390625 disk.drbd1260.in‚0! disk.nvme0n1.in‚ 26914.2578125$ memory_percent‚99.04162883758545 network.sent‚ 1.7090909e+07 disk.sda2.out‚0 disk.dm-1.out‚ 328.69921875 disk.sda.out‚ 328.69921875 disk.sda3.in‚ 0.00390625' disk.nvme0n1.out‚259500.34912109375" proc.memory.rssMB‚ 3125.9609375( proc.memory.percent‚19.0793514251709 network.recv‚ 3.380071e+06&=È‘Ø ‚(:† јèÔˆïþx gpu.0.gpu‚53.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚45.0 gpu.0.powerWatts‚263.438' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6876 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öo¨Ù ‚(:… à˜èÔÌ©¡w gpu.0.gpu‚26.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚46.0 gpu.0.powerWatts‚248.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.693 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬á««žÚ ‚(:“ à˜èÔµÄæx disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 26914.2578125$ disk.dm-0.out‚259595.72021484375$ memory_percent‚99.08883571624756* proc.memory.percent‚19.126534461975098 network.sent‚ 1.7095756e+07 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.61328125 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 329.1640625 network.recv‚ 3.383126e+06! disk.nvme0n1.in‚ 26914.2578125 disk.sda1.in‚0) disk.nvme0c0n1.out‚259595.72021484375 disk.dm-0.in‚ 26914.2578125 disk.sda.out‚ 329.1640625( proc.memory.availableMB‚ 149.28515625 cpu‚25.211291903301596$ disk./.usageGB‚1161.870506286621' disk.nvme0n1.out‚259595.72021484375 disk.sda1.out‚0 disk.dm-1.out‚ 329.1640625 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625# proc.memory.rssMB‚ 3133.69140625 proc.cpu.threads‚17) disk./.usagePercent‚77.97410455118316"œÛ ‚(:‘ ï˜èÔÄËãx cpu‚25.21127652269902 disk.sda3.in‚ 0.00390625 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 330.37890625& disk.nvme0n1.out‚259788.6279296875) proc.memory.percent‚19.16182041168213 proc.cpu.threads‚17) disk./.usagePercent‚77.97410096707767( disk.nvme0c0n1.out‚259788.6279296875 disk.dm-1.out‚ 330.37890625! disk.nvme0n1.in‚ 26914.2578125 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚99.12407398223877# proc.memory.rssMB‚ 3139.47265625 network.sent‚ 1.7102154e+07% disk./.usageGB‚1161.8704528808594# disk.nvme0c0n1.in‚ 26914.2578125 disk.dm-0.in‚ 26914.2578125# disk.dm-0.out‚259788.6279296875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6171875 disk.sda1.in‚0 network.recv‚ 3.388395e+06 disk.sda3.out‚ 330.37890625 disk.sda1.out‚0( proc.memory.availableMB‚ 143.51171875Ç>©‘Ü ‚(:† ï˜èÔÌóÞy gpu.0.gpu‚11.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚192.826' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.5652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01p@F’Ý ‚(:‡ þ˜èÔ¶‚µw gpu.0.gpu‚96.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚41.0 gpu.0.powerWatts‚130.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0604 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%fõ’˜Þ ‚(: þ˜èÔx network.sent‚ 1.7107089e+07 network.recv‚ 3.391578e+06! disk.nvme0n1.in‚ 26914.2578125 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 26914.2578125$ disk./.usageGB‚1161.770351409912 disk.sda.out‚ 330.8203125 disk.sda3.out‚ 330.8203125( disk.nvme0c0n1.out‚260246.7099609375 disk.dm-0.in‚ 26914.2578125' proc.memory.availableMB‚ 140.6171875* proc.memory.percent‚19.177985191345215 cpu‚25.211263279013455) disk./.usagePercent‚77.96738307337168 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6171875& disk.nvme0n1.out‚260246.7099609375 disk.dm-1.in‚ 0.00390625# proc.memory.rssMB‚ 3142.12109375 proc.cpu.threads‚17 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625# disk.dm-0.out‚260246.7099609375 disk.dm-1.out‚ 330.8203125# memory_percent‚99.1417407989502 disk.sda2.out‚0qå¬Êß ‚(:… ™èԃܜx gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nqª”‘à ‚(:† ™èÔžšûx disk.sda3.out‚ 331.12890625' proc.memory.availableMB‚ 30.44921875 network.sent‚ 1.7113348e+07 network.recv‚ 3.39626e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 27034.5625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 331.12890625$ memory_percent‚99.81415271759033 disk.drbd1260.out‚ 1.62109375% disk.nvme0n1.out‚260591.181640625' disk.nvme0c0n1.out‚260591.181640625" disk.dm-0.out‚260591.181640625# proc.memory.rssMB‚ 3252.13671875* proc.memory.percent‚19.849467277526855) disk./.usagePercent‚77.97266911693868 disk.nvme0n1.in‚ 27034.5625 disk.dm-0.in‚ 27034.5625 disk.sda3.in‚ 0.00390625 cpu‚25.216470066744048 proc.cpu.threads‚17% disk./.usageGB‚1161.8491172790527 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.00390625 disk.sda.out‚ 331.12890625¶Xäéá ‚(:‘ œ™èÔúÂÜw gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚231.368' gpu.0.enforcedPowerLimitWXV,ý³atts‚500.0) gpu.0.powerPercent‚46.273599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ɉ%â ‚(:„ œ™èÔÍÉ–x disk.nvme0n1.in‚ 27034.5625 disk.sda2.out‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 27034.5625 disk.dm-0.in‚ 27034.5625 disk.sda.out‚ 331.48046875# proc.memory.rssMB‚ 3255.22265625 disk.sda2.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚260961.509765625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 331.48046875 disk.drbd1260.in‚0$ memory_percent‚99.83108043670654 proc.cpu.threads‚17 network.recv‚ 3.399135e+06 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 331.48046875' proc.memory.availableMB‚ 27.67578125) proc.memory.percent‚19.86830234527588 network.sent‚ 1.7117979e+07) disk./.usagePercent‚77.97638020215501% disk./.usageGB‚1161.9044151306152% disk.nvme0n1.out‚260961.509765625" disk.dm-0.out‚260961.509765625 disk.drbd1260.out‚ 1.62109375 cpu‚25.2208037415308/x!ã ‚(:ö «™èÔªy disk.nvme0n1.in‚ 27034.5625 disk.sda1.in‚0# proc.memory.rssMB‚ 3257.36328125 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚331.875' disk.nvme0n1.out‚261249.42333984375 cpu‚25.224570681280838 network.recv‚ 3.404328e+06) disk./.usagePercent‚77.97314401091468 disk.nvme0c0n1.in‚ 27034.5625) disk.nvme0c0n1.out‚261249.42333984375# memory_percent‚99.8438835144043% proc.memory.availableMB‚ 25.578125* proc.memory.percent‚19.881367683410645 proc.cpu.threads‚17 disk.dm-0.in‚ 27034.5625$ disk.dm-0.out‚261249.42333984375 disk.sda.in‚ 0.00390625 disk.sda.out‚331.875 disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 network.sent‚ 1.7124015e+07% disk./.usageGB‚1161.8561935424805 disk.dm-1.out‚331.875 disk.drbd1260.in‚0 disk.drbd1260.out‚1.625:ñÕ–œä ‚(:‘ «™èÔ•©‡z gpu.0.gpu‚41.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚158.004' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.600799999999996 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fŸš›å ‚(: º™èÔºÚx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚111.504' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.300800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°äŽæ ‚(:ƒ º™èÔÔøÄy disk.drbd1260.out‚1.625$ memory_percent‚99.85051155090332" proc.memory.rssMB‚ 3258.4453125 cpu‚25.2287625207122 proc.cpu.threads‚17$ disk./.usageGB‚1161.860939025879 disk.nvme0c0n1.in‚ 27034.5625) disk.nvme0c0n1.out‚261494.97021484375 disk.dm-1.in‚ 0.00390625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 332.44140625 disk.sda3.out‚ 332.44140625 disk.drbd1260.in‚0 network.sent‚ 1.7132302e+07 disk.dm-0.in‚ 27034.5625$ disk.dm-0.out‚261494.97021484375 disk.dm-1.out‚ 332.44140625' disk.nvme0n1.out‚261494.97021484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 27034.5625 disk.sda1.in‚0& proc.memory.availableMB‚ 24.4921875* proc.memory.percent‚19.887971878051758 network.recv‚ 3.414261e+06) disk./.usagePercent‚77.97346248428728 disk.sda3.in‚ 0.00390625Êêm¦œç ‚(:‘ É™èÔºûýx gpu.0.gpu‚31.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.806' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.361200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉOƒ›è ‚(: É™èÔŸãÂz disk.sda3.in‚ 0.00390625 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.7138587e+07$ disk.dm-0.out‚261590.93994140625 disk.sda3.out‚ 332.69921875' disk.nvme0n1.out‚261590.93994140625# memory_percent‚99.8544692993164$ proc.memory.availableMB‚23.84375 proc.memory.rssMB‚ 3259.09375 cpu‚25.232417898696813 network.recv‚ 3.418819e+06$ disk.nvme0c0n1.in‚27151.97265625 disk.dm-0.in‚27151.97265625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6328125" disk.nvme0n1.in‚27151.97265625 disk.sda1.in‚0 disk.sda1.out‚0( disk./.usagePercent‚77.9733997624414 disk.sda.out‚ 332.69921875* proc.memory.percent‚19.891929626464844 proc.cpu.threads‚17% disk./.usageGB‚1161.8600044250488) disk.nvme0c0n1.out‚261590.93994140625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 332.69921875 disk.sda.in‚ 0.00390625VI¡4é / _wandbruntime‚10922  _runtime‚10922Î…ý9z*ê ôT YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚81135720 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚939045182679312 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1532 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚14740011244032 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1221 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚14742719803392 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚976927999379456 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2706586280 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2708559360 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚870851526114304 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2729917096 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3719 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2090 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 279744457 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2732853760 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚976925266525696 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 128887296 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚976927999379456 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚870851566831616 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1532 _step‚492 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 279765170 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚962185255281664 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 151587405 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚962185279576064 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29392819712 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚205 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚14716863959482 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚14714157373202 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚962185279576064 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 151566697 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚20281353465344 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚953759340052514 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1532 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3719 trainer/global_step‚6044 _runtime‚10922.99182081 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 136333627 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 279744457 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚891132879579648 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 136333422 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 33395048448 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 33395048448 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2187 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚962185255281664 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 128177760 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 4586471424 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2732853760 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚14740011244032 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚891133049184256 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708559360 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32002 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880505161 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2187 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 128177765 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32002 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3Ðí.*ë ùT YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 25702182912 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32002 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 28880505161 RIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚2187 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 128177765 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32002 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚3 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚81135720 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚939045182679312 MDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚1532 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚14740011244032 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1221 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚14742719803392 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚976927999379456 YJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 3896742400 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 2706586280 ^ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 2708559360 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚870851526114304 WHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 2729917096 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚3719 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚23330816 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚40717312 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3921 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚2090 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 279744457 TEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 2732853760 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚976925266525696 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 128887296 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚976927999379456 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚870851566831616 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚1532 _step‚492 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 279765170 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚962185255281664 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 151587405 `QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚5 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚962185279576064 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 29392819712 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚205 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚14716863959482 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚24294400 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚14714157373202 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚962185279576064 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 151566697 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚20281353465344 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚953759340052514 PGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚1532 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚3719 trainer/global_step‚6044 _runtime‚10922.99182081 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 136333627 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 279744457 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚891132879579648 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 136333422 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 33395048448 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 33395048448 MDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚2187 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚962185255281664 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 128177760 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 4586471424 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚24294400 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 2732853760 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚14740011244032 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚891133049184256 [LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 2708559360ìø—0ê4ì / _wandbruntime‚10924  _runtime‚109246) …í ÿR ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚891228620508672 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚2187 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚962292916085760 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 128193325 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚891228790113280 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 4509787136 ^ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚953862635145198 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 128193320 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3719 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚977036186467328 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚870946757182464 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 33395048448 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 33395048448 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3921 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1532 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚81148065 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1221 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚20282032930816 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚962292916085760 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32002 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚870946716465152 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚14714706311126 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6044 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32002 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚20281904043520 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2729917096 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2708559360 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚939145198916976 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚14740561822208 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 151575427 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚953859905228102 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚939145222247792 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507766904 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3719 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 279789460 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚14740561822208 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1532 % _timestamp‚1.7884807174563057e+09 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚962292891791360 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1532 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2732853760 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3719 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚3719 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚977036186467328 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚870946757182464 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚23330816 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 33395048448 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 33395048448 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3921 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚1532 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚81148065 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1221 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚20282032930816 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚962292916085760 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32002 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚870946716465152 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚14714706311126 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6044 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚5 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32002 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚24294400 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚20281904043520 UFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 2729917096 YJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 2708559360 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚939145198916976 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚14740561822208 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 151575427 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚953859905228102 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚939145222247792 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 4507766904 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚3719 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 279789460 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚14740561822208 WHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 3896742400 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 NEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚1532 % _timestamp‚1.7884807174563057e+09 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚962292891791360 PGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚1532 RCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 2732853760 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚3719 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4903954565525055 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 8"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.346569299697876 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948438405990601 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48464909195899963 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21432748436927795 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.212301567196846 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18165312707424164 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10211053490638733 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4309028089046478 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.49537038803100586 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2341507375240326 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.647537350654602 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_7_of_7‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4354938566684723 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018033508211374283 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 *jet/accuracy_1_of_3‚0.8558758497238159 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991741955280304 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18290191888809204 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3194444477558136 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4364054203033447 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18163788318634033 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2234848439693451 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3651738166809082 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1828886717557907 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6943857073783875 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3147861659526825 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19514332711696625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 'particle/f_score‚0.3793051540851593 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17290444672107697 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4858924448490143 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.051360391080379486 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7651514410972595 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9915164709091187 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4934459626674652 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980685710906982 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23978763818740845 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4595537781715393 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.45777004957199097 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2298833578824997 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26018521189689636 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 _step‚494 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27450698614120483 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47759896516799927 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25061067938804626 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9950501918792725 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3447834849357605 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.159465029835701 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48254987597465515 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905416369438171 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_1_of_6‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9314422011375427 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23446325957775116 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt1SRqqt2FBt/event_purity‚0.04545454680919647 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18850575387477875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5041859149932861 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47206154465675354 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.592650294303894 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10974246263504028 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21062971651554108 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20711782574653625 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013353489339351654 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.929894208908081 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49990612268447876 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_5‚0.47784826159477234 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 *loss/val_total_loss‚44.473580778189316 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9891800880432129 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936014413833618 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0062289759516716 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02058883011341095 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04465408995747566 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2260802537202835 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 epoch‚4 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4663601517677307 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_6‚0.01923076994717121 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24397259950637817 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24703878164291382 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926905632019043 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15421849489212036 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004079694859683514 jet/accuracy_1_of_7‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.24126608669757843 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.4324706792831421 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18158045411109924 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03381313011050224 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18072879314422607 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.35475316643714905 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ,jet/accuracy_3_of_3‚0.021287454292178154 particle/accuracy_2_of_2‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37342044711112976 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9236111640930176 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4436853229999542 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012767874635756016 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_6‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41447463631629944 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08476337045431137 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18040771782398224 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3444443643093109 _runtime‚10949.389039433 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48747822642326355 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 @&Purity/*FçHÐWTRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.991738498210907 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3712962567806244 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43444696068763733 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6051587462425232 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4612068831920624 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36329346895217896 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3779761791229248 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6089285612106323 *jet/accuracy_1_of_4‚0.9024466872215271 particle/accuracy_3_of_4‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 +particle/specificity‚0.9509583115577698 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.413194477558136 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49282944202423096 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41360729932785034 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987751245498657 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27897897362709045 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2295418232679367 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5937500596046448 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9917170405387878 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 trainer/global_step‚6044 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023156682029366493 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1832180917263031 particle/accuracy_2_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9991645812988281 particle/accuracy_3_of_6‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02298850566148758 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9953389167785645 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 particle/accuracy_5_of_5‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4614197313785553 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.45750999450683594 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5716705322265625 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17296604812145233 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9948377013206482 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.748058557510376 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578946709632874 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4632192850112915 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38827577233314514 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.18333332240581512 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994879961013794 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4616255462169647 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5492216348648071 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12498048692941666 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17117875814437866 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4253876209259033 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9905426502227783 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6105300188064575 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16689029335975647 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3434675931930542 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4580400884151459 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0030738224741071463 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9991645812988281 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4868637025356293 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.811927318572998 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5389624238014221 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 % _timestamp‚1.7884807426310372e+09 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5420891642570496 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0034486018121242523 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_2_of_7‚0.5 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02124415710568428 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8487411141395569 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5100907683372498 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2430555522441864 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9976752400398254 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5483602285385132 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5368332266807556 ,jet/accuracy_4_of_5‚0.015322580002248287 particle/accuracy_4_of_4‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5027983784675598 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18403148651123047 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08035920560359955 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4292563796043396 +jet/accuracy_2_of_2‚0.10743417590856552 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.34508925676345825 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5162837505340576 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23309554159641266 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1507936418056488 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5483230948448181 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 jet/accuracy_1_of_6‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29232341051101685 particle/accuracy_5_of_6‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2569412589073181 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3227308392524719 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_1_of_4‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9181286692619324 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4561585485935211 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3919973373413086 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1533333659172058 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15970346331596375 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3694043457508087 *jet/accuracy_3_of_4‚0.0656484067440033 particle/accuracy_1_of_3‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2916666567325592 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4872901737689972 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515007019043 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5428701639175415 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2323412448167801 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.370593786239624 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_5_of_5‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5700565576553345 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0401785708963871 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10636840760707855 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5806657075881958 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924587607383728 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15773437917232513 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5295966863632202 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2883838415145874 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.021495815366506577 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4000401198863983 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18617546558380127 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.003117913845926523 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21462760865688324 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1906130164861679 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4858924448490143 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42490020394325256 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5256410241127014 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18340009450912476 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1828886717557907 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348484396934509 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17940127849578857 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21959459781646729 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_5‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3745097815990448 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3863018751144409 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5488911867141724 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2794986665248871 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_5‚0.9418330788612366 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9861111640930176 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2534048557281494 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1843860149383545 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5753591060638428 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629629850387573 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4830826222896576 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008112874813377857 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5545976758003235 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3194444477558136 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.223908931016922 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 particle/accuracy_3_of_5‚1 jet/accuracy_6_of_7‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21350574493408203 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2430555522441864 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2271273285150528 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.00588212488219142 particle/accuracy_6_of_6‚1 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991154074668884 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,particle/sensitivity‚0.29632794857025146 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0ùÕ„›+ó Ĥ ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5716705322265625 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17296604812145233 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9948377013206482 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_2‚0.748058557510376 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578946709632874 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4632192850112915 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38827577233314514 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.18333332240581512 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9994879961013794 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4616255462169647 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5492216348648071 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12498048692941666 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17117875814437866 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4253876209259033 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9905426502227783 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6105300188064575 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16689029335975647 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3434675931930542 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4580400884151459 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0030738224741071463 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9991645812988281 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4868637025356293 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.811927318572998 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5389624238014221 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 % _timestamp‚1.7884807426310372e+09 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5420891642570496 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0034486018121242523 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_2_of_7‚0.5 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02124415710568428 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8487411141395569 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5100907683372498 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2430555522441864 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9976752400398254 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5483602285385132 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5368332266807556 ,jet/accuracy_4_of_5‚0.015322580002248287 particle/accuracy_4_of_4‚1 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5027983784675598 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18403148651123047 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08035920560359955 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4292563796043396 +jet/accuracy_2_of_2‚0.10743417590856552 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.34508925676345825 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5162837505340576 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.23309554159641266 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1507936418056488 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ( Purity/1FRt4SRqqt0FBt/FRt_purity‚0.5 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5483230948448181 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 jet/accuracy_1_of_6‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29232341051101685 particle/accuracy_5_of_6‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2569412589073181 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3227308392524719 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_1_of_4‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9181286692619324 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4561585485935211 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.3919973373413086 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1533333659172058 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15970346331596375 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3694043457508087 *jet/accuracy_3_of_4‚0.0656484067440033 particle/accuracy_1_of_3‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2916666567325592 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4872901737689972 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515007019043 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5428701639175415 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2323412448167801 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.370593786239624 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_5_of_5‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5700565576553345 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0401785708963871 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10636840760707855 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5806657075881958 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9924587607383728 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15773437917232513 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5295966863632202 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2883838415145874 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.021495815366506577 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4000401198863983 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18617546558380127 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.003117913845926523 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21462760865688324 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1906130164861679 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.012152778916060925 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Ý•°fPurity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4858924448490143 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42490020394325256 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.5256410241127014 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18340009450912476 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1828886717557907 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7348484396934509 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17940127849578857 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.21959459781646729 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_5‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3745097815990448 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3863018751144409 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5488911867141724 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2794986665248871 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_5‚0.9418330788612366 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9861111640930176 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2534048557281494 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1843860149383545 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5753591060638428 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9629629850387573 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4830826222896576 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008112874813377857 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5545976758003235 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3194444477558136 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.223908931016922 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 particle/accuracy_3_of_5‚1 jet/accuracy_6_of_7‚0 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21350574493408203 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2430555522441864 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2271273285150528 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.00588212488219142 particle/accuracy_6_of_6‚1 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991154074668884 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,particle/sensitivity‚0.29632794857025146 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.017641976475715637 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9897977113723755 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5466687083244324 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5447916388511658 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9995331764221191 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.049783557653427124 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5995835661888123 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4903954565525055 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.3125 8"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.346569299697876 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948438405990601 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48464909195899963 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21432748436927795 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.212301567196846 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18165312707424164 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10211053490638733 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4309028089046478 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.49537038803100586 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2341507375240326 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.647537350654602 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_7_of_7‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4354938566684723 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018033508211374283 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 *jet/accuracy_1_of_3‚0.8558758497238159 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991741955280304 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18290191888809204 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3194444477558136 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4364054203033447 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18163788318634033 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3750000298023224 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2234848439693451 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3651738166809082 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1828886717557907 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6943857073783875 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3147861659526825 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19514332711696625 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 'particle/f_score‚0.3793051540851593 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17290444672107697 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4858924448490143 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.051360391080379486 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7651514410972595 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9915164709091187 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4934459626674652 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980685710906982 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23978763818740845 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4595537781715393 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.45777004957199097 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2298833578824997 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26018521189689636 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 _step‚494 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27450698614120483 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47759896516799927 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25061067938804626 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9950501918792725 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3447834849357605 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.159465029835701 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48254987597465515 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905416369438171 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_1_of_6‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9314422011375427 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23446325957775116 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt1SRqqt2FBt/event_purity‚0.04545454680919647 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18850575387477875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5041859149932861 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47206154465675354 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.592650294303894 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10974246263504028 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.21062971651554108 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20711782574653625 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013353489339351654 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.929894208908081 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49990612268447876 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_5‚0.47784826159477234 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 *loss/val_total_loss‚44.473580778189316 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9891800880432129 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/1FRt4SRqqt*FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936014413833618 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0062289759516716 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02058883011341095 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04465408995747566 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2260802537202835 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 epoch‚4 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4663601517677307 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_6‚0.01923076994717121 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24397259950637817 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24703878164291382 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9926905632019043 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15421849489212036 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004079694859683514 jet/accuracy_1_of_7‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.24126608669757843 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.4324706792831421 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18158045411109924 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03381313011050224 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.18072879314422607 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.35475316643714905 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ,jet/accuracy_3_of_3‚0.021287454292178154 particle/accuracy_2_of_2‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37342044711112976 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9629628658294678 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9236111640930176 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4436853229999542 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012767874635756016 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_6‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41447463631629944 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08476337045431137 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18040771782398224 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3444443643093109 _runtime‚10949.389039433 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.48747822642326355 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.991738498210907 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3712962567806244 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43444696068763733 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6051587462425232 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4612068831920624 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36329346895217896 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3779761791229248 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6089285612106323 *jet/accuracy_1_of_4‚0.9024466872215271 particle/accuracy_3_of_4‚1 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 +particle/specificity‚0.9509583115577698 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.413194477558136 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49282944202423096 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.41360729932785034 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987751245498657 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27897897362709045 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2295418232679367 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5937500596046448 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9917170405387878 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9984127283096313 jet/accuracy_3_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 trainer/global_step‚6044 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023156682029366493 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1832180917263031 particle/accuracy_2_of_6‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9991645812988281 particle/accuracy_3_of_6‚1 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02298850566148758 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9953389167785645 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 particle/accuracy_5_of_5‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4614197313785553 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.45750999450683594îskpô ‚(:… ç™èÔ߀šx gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚34.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚» }–õ ‚(:‹ ç™èÔÑÒ÷z disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚77.98026537249466$ disk.nvme0c0n1.in‚27151.97265625& disk.nvme0c0n1.out‚262429.30859375! disk.dm-0.out‚262429.30859375 disk.dm-1.out‚ 333.65234375 disk.sda.in‚ 0.00390625 disk.sda.out‚ 333.65234375 disk.drbd1260.in‚0 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 333.65234375 disk.drbd1260.out‚ 1.63671875 disk.sda2.out‚0% proc.memory.availableMB‚ 19.890625 proc.memory.rssMB‚ 3262.40625 cpu‚25.239977520586145 network.sent‚ 1.7253736e+07% disk./.usageGB‚1161.9623069763184$ memory_percent‚99.87859725952148* proc.memory.percent‚19.912147521972656 network.recv‚ 3.430585e+06 disk.dm-0.in‚27151.97265625" disk.nvme0n1.in‚27151.97265625$ disk.nvme0n1.out‚262429.30859375s¯›÷ö ‚(:„ ö™èÔ¿ˆ˜x gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûP¤“÷ ‚(:ˆ ö™èÔ”Ûèx$ memory_percent‚80.37958145141602# proc.memory.rssMB‚ 5245.17578125 network.sent‚ 1.7376287e+07" disk.nvme0n1.out‚ 266499.390625 disk.sda2.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚27151.97265625 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 334.14453125 disk.sda.in‚ 0.00390625 network.recv‚ 3.43604e+06) disk./.usagePercent‚77.99022944173481% disk./.usageGB‚1162.1107788085938 disk.drbd1260.in‚0" disk.nvme0n1.in‚27151.97265625 disk.sda1.in‚0$ disk.nvme0c0n1.out‚ 266499.390625 disk.sda3.out‚ 334.14453125' proc.memory.availableMB‚ 3214.609375) proc.memory.percent‚32.01401233673096 cpu‚25.239729929966288 proc.cpu.threads‚17 disk.drbd1260.out‚ 1.63671875 disk.sda2.out‚0 disk.dm-0.in‚27151.97265625 disk.dm-0.out‚ 266499.390625 disk.sda.out‚ 334.14453125 disk.sda3.in‚ 0.00390625„4µ4ø / _wandbruntime‚10977  _runtime‚10977ÀaÉOÊù Ä trainer/global_step‚6049 % _timestamp‚1.7884807708264537e+09 _runtime‚10977.583416063 _step‚495 ' lr-AdamW/pg1‚0.00038215178594139937 ' lr-AdamW/pg2‚0.00038215178594139937ÁæiÏú É ' lr-AdamW/pg1‚0.00038215178594139937 ' lr-AdamW/pg2‚0.00038215178594139937 trainer/global_step‚6049 % _timestamp‚1.7884807708264537e+09 _runtime‚10977.583416063 _step‚495ïå–04û / _wandbruntime‚10977  _runtime‚109773DÆlÁ ü P H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚964307227937280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚964307227937280 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151785163 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚893043949481984 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151811503 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128477295 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280262458 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14760076243968 _step‚496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14756465583104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979067330572800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 % _timestamp‚1.7884807708283648e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81359789 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979063693520384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚941046750301968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100072448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55316274 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14760076243968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128477301 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20302525946368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚941046724874000 trainer/global_step‚6049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55315762 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136676063 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48869376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979063693520384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚941046750301968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100072448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55316274 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14760076243968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128477301 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20302525946368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚941046724874000 trainer/global_step‚6049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55315762 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136676063 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚48869376 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151811503 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128477295 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280262458 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14760076243968 _step‚496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14756465583104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979067330572800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 % _timestamp‚1.7884807708283648e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81359789 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14732976846722 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280307449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884807719304354e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚979223518626816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280333821 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20305940754944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚964461005496320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55324534 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚893186705632768 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128497719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151809730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81372594 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚872880662721536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20305991708160 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280307449 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136697123 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14762486738944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280333821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128497719 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚10978.688077751 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151836096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚964461005496320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚979223518626816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81372589 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚955933665052718 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6049 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚941197054115136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚979219881561088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14758876064768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚941197079543104 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚964461031887872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚893186603476480 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 _step‚497 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14762486738944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424+©`¥"€!œN BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128497719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151809730 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81372594 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚872880662721536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20305991708160 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280307449 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136697123 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14762486738944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280333821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128497719 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 _runtime‚10978.688077751 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151836096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚964461005496320 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚979223518626816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81372589 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚955933665052718 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6049 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚941197054115136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚979219881561088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14758876064768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚941197079543104 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚964461031887872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚893186603476480 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ZMDevicenWÀStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 _step‚497 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14762486738944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128497725 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14736585509614 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14732976846722 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280307449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884807719304354e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚979223518626816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280333821 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20305940754944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚964461005496320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55324534 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚893186705632768ñðz©4!/ _wandbruntime‚10978  _runtime‚10978Ye6T‚!Î .loss/FRt2/assignment_loss‚2.94054743523399 /loss/FRt4/assignment_loss‚3.006744289036953 /loss/FRt3/detection_loss‚3.1791942630495345 0loss/SRqqt4/detection_loss‚2.015250715689781 $ _timestamp‚1.788480771933342e+09 /loss/FRt4/detection_loss‚2.8995396115563135 .loss/FBt1/detection_loss‚2.414635880635335 -loss/FBt3/detection_loss‚2.73862255414327 %loss/total_loss‚45.68058481670718 /loss/FRt3/assignment_loss‚4.675764765058245 2loss/SRqqt1/assignment_loss‚0.8643169824795153 /loss/FBt2/detection_loss‚0.8911206893694515 .loss/FBt4/detection_loss‚1.943882339530521 _runtime‚10978.688957409 1loss/SRqqt3/assignment_loss‚2.076138405238881 0loss/FBt1/assignment_loss‚0.2787474741853657 .loss/FRt1/detection_loss‚1.546801257917257 1loss/SRqqt2/assignment_loss‚2.138213915893665 1loss/FBt4/assignment_loss‚0.42893025227163484 .loss/FRt2/detection_loss‚2.096666549642881 1loss/SRqqt1/detection_loss‚0.8154388232013354 1loss/SRqqt2/detection_loss‚2.4449786131198588 0loss/SRqqt3/detection_loss‚2.352981123854132 epoch‚5 2loss/SRqqt4/assignment_loss‚1.5189771105845769 1loss/FBt2/assignment_loss‚0.13755193219090311 1loss/FBt3/assignment_loss‚0.46021549170060705 _step‚498 trainer/global_step‚6049 0loss/FRt1/assignment_loss‚1.8153243411231685 ø\Yƒ!Ó 1loss/SRqqt3/assignment_loss‚2.076138405238881 0loss/FBt1/assignment_loss‚0.2787474741853657 .loss/FRt1/detection_loss‚1.546801257917257 1loss/SRqqt2/assignment_loss‚2.138213915893665 1loss/FBt4/assignment_loss‚0.42893025227163484 .loss/FRt2/detection_loss‚2.096666549642881 1loss/SRqqt1/detection_loss‚0.8154388232013354 1loss/SRqqt2/detection_loss‚2.4449786131198588 0loss/SRqqt3/detection_loss‚2.352981123854132 epoch‚5 2loss/SRqqt4/assignment_loss‚1.5189771105845769 1loss/FBt2/assignment_loss‚0.13755193219090311 1loss/FBt3/assignment_loss‚0.46021549170060705 _step‚498 trainer/global_step‚6049 0loss/FRt1/assignment_loss‚1.8153243411231685 .loss/FRt2/assignment_loss‚2.94054743523399 /loss/FRt4/assignment_loss‚3.006744289036953 /loss/FRt3/detection_loss‚3.1791942630495345 0loss/SRqqt4/detection_loss‚2.015250715689781 $ _timestamp‚1.788480771933342e+09 /loss/FRt4/detection_loss‚2.8995396115563135 .loss/FBt1/detection_loss‚2.414635880635335 -loss/FBt3/detection_loss‚2.73862255414327 %loss/total_loss‚45.68058481670718 /loss/FRt3/assignment_loss‚4.675764765058245 2loss/SRqqt1/assignment_loss‚0.8643169824795153 /loss/FBt2/detection_loss‚0.8911206893694515 .loss/FBt4/detection_loss‚1.943882339530521 _runtime‚10978.688957409òÆ3¸?Ÿ„!‚(:” …šèԀĿy disk.sda1.out‚0$ disk.nvme0c0n1.in‚27300.60546875 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 334.5078125 disk.sda3.in‚ 0.00390625# proc.memory.rssMB‚ 3263.34765625$ disk.dm-0.out‚272975.95849609375 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-0.in‚27300.60546875 disk.sda.out‚ 334.5078125 disk.drbd1260.out‚1.640625$ memory_percent‚68.27967166900635 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 334.5078125 disk.drbd1260.in‚0 cpu‚25.2385218711913 network.sent‚ 1.7383488e+07 network.recv‚ 3.441634e+06% disk./.usageGB‚1163.5778045654297" disk.nvme0n1.in‚27300.60546875) disk.nvme0c0n1.out‚272975.95849609375 disk.sda.in‚ 0.00390625) proc.memory.availableMB‚ 5197.05859375* proc.memory.percent‚19.917893409729004) disk./.usagePercent‚78.08868277119268' disk.nvme0n1.out‚272975.95849609375ñ´©“’…!‚(:‡ …šèÔÂÖ—z gpu.0.gpu‚59.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚145.754' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.1508 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M1×’†!‚(:‡ ”šèÔÙö¬w gpu.0.gpu‚86.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚234.838' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.9676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0щ›dŸ‡!‚(:” ”šèÔª¢™x$ disk.nvme0c0n1.in‚27300.60546875 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 334.8515625 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚68.28474998474121( proc.memory.availableMB‚ 5196.2265625$ disk.dm-0.out‚273129.26318359375 disk.drbd1260.out‚1.640625" disk.nvme0n1.in‚27300.60546875' disk.nvme0n1.out‚273129.26318359375* proc.memory.percent‚19.919347763061523 network.sent‚ 1.7490816e+07% disk./.usageGB‚1163.8278045654297 disk.dm-0.in‚27300.60546875 disk.sda.in‚ 0.00390625 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0" proc.memory.rssMB‚ 3263.5859375 cpu‚25.238475829839327) disk./.usagePercent‚78.10546048095665) disk.nvme0c0n1.out‚273129.26318359375 disk.dm-1.out‚ 334.8515625 disk.sda3.out‚ 334.8515625 disk.drbd1260.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 3.447129e+067åíØ’ˆ!‚(:‡ £šèÔÈ‚†x gpu.0.gpu‚91.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚44.0 gpu.0.powerWatts‚234.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.9654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰‚ó"¢‰!‚(:— £šèÔ¹ÿèx) disk.nvme0c0n1.out‚273500.16845703125 disk.drbd1260.out‚ 1.64453125$ memory_percent‚68.28618049621582# proc.memory.rssMB‚ 3263.97265625' disk.nvme0n1.out‚273500.16845703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 335.36328125 disk.sda.in‚ 0.00390625( proc.memory.availableMB‚ 5195.9921875 proc.cpu.threads‚17( disk./.usagePercent‚78.1054579208813 disk.dm-0.in‚27300.60546875$ disk.dm-0.out‚273500.16845703125 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 335.36328125 disk.drbd1260.in‚0) proc.memory.percent‚19.92170810699463 cpu‚25.238488435369575 network.sent‚ 1.7510425e+07 network.recv‚ 3.478371e+06$ disk./.usageGB‚1163.827766418457" disk.nvme0n1.in‚27300.60546875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚27300.60546875 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 335.36328125±Å|O–Š!‚(:‹ ²šèÔô€¥y) disk./.usagePercent‚78.10388296253066 disk.drbd1260.out‚ 1.64453125" disk.nvme0n1.in‚27300.60546875$ disk.nvme0c0n1.in‚27300.60546875 disk.sda3.in‚ 0.00390625* proc.memory.percent‚19.922280311584473 cpu‚25.238456864339085 network.sent‚ 1.7515509e+07 disk.sda1.out‚0) disk.nvme0c0n1.out‚273846.84814453125 disk.dm-0.in‚27300.60546875$ disk.dm-0.out‚273846.84814453125 disk.sda.in‚ 0.00390625 disk.sda.out‚335.875 network.recv‚ 3.481753e+06$ disk./.usageGB‚1163.804298400879 disk.drbd1260.in‚0 disk.sda1.in‚0 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚335.875) proc.memory.availableMB‚ 5195.90234375 proc.cpu.threads‚17' disk.nvme0n1.out‚273846.84814453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚335.875$ memory_percent‚68.28672885894775# proc.memory.rssMB‚ 3264.06640625ÛÖØ‹!‚(:’ ²šèÔÍ°z gpu.0.gpu‚53.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚154.246' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.849200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l‡²˜Œ!‚(: ÁšèÔö¿»y disk.sda.in‚ 0.00390625 disk.sda3.out‚ 336.11328125" disk.nvme0n1.in‚27425.56640625 disk.sda2.in‚0$ disk.nvme0c0n1.in‚27425.56640625 proc.cpu.threads‚17% disk./.usageGB‚1164.3063163757324 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 336.11328125 disk.sda2.out‚0# proc.memory.rssMB‚ 3264.44140625* proc.memory.percent‚19.924569129943848 network.sent‚ 1.753355e+07) disk./.usagePercent‚78.13757381004422 disk.sda3.in‚ 0.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.6484375# disk.nvme0n1.out‚274226.7734375 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 3.506188e+06 disk.dm-0.in‚27425.56640625% disk.nvme0c0n1.out‚274226.5234375$ memory_percent‚68.30265522003174) proc.memory.availableMB‚ 5193.29296875 cpu‚25.238449128085627 disk.dm-0.out‚274225.6484375 disk.dm-1.out‚ 336.11328125•¬"ˆ!‚(:„ ÁšèÔ÷Úz gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚36.0 gpu.0.powerWatts‚96.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2168 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0† þ€Ž!‚(:’ КèÔ”îåy gpu.0.gpu‚50.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âíÒG›!‚(: КèÔ²ï£z disk.sda1.in‚0& disk.nvme0c0n1.out‚274669.19140625 disk.drbd1260.out‚ 1.6484375" disk.nvme0n1.in‚27425.56640625 disk.sda2.out‚0$ disk.nvme0c0n1.in‚27425.56640625$ memory_percent‚68.29018592834473* proc.memory.percent‚19.925618171691895 proc.cpu.threads‚17% disk./.usageGB‚1164.3277854919434 disk.dm-1.in‚ 0.00390625 disk.sda.out‚ 336.44140625 disk.sda3.in‚ 0.00390625 disk.sda1.out‚0# proc.memory.rssMB‚ 3264.61328125 disk.dm-1.out‚ 336.44140625 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 336.44140625$ disk.nvme0n1.out‚274669.19140625 disk.sda2.in‚0( proc.memory.availableMB‚ 5195.3359375 cpu‚25.238483350273402 network.sent‚ 1.7536514e+07 network.recv‚ 3.509695e+06) disk./.usagePercent‚78.13901462044691 disk.dm-0.in‚27425.56640625! disk.dm-0.out‚274669.19140625 disk.drbd1260.in‚0[´[N”!‚(:‰ ßšèÔ€šÛy disk.sda.out‚ 336.96875 disk.sda3.in‚ 0.00390625 disk.drbd1260.out‚1.65625 disk.sda2.out‚0% proc.memory.availableMB‚ 9.8203125# proc.memory.rssMB‚ 3264.77734375) disk.nvme0c0n1.out‚274991.92626953125 disk.sda.in‚ 0.00390625' disk.nvme0n1.out‚274991.92626953125 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.514676e+06% disk./.usageGB‚1164.3275527954102 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda1.out‚0) proc.memory.percent‚19.92661952972412 cpu‚25.23829990926601 network.sent‚ 1.7543034e+07) disk./.usagePercent‚78.13899900398734$ disk.nvme0c0n1.in‚28121.34765625$ disk.dm-0.out‚274991.92626953125 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 336.96875" disk.nvme0n1.in‚28121.34765625$ memory_percent‚99.94006156921387 disk.dm-0.in‚28121.34765625 disk.dm-1.out‚ 336.96875uv¦d‘!‚(:’ ßšèÔŸ»ž¤ gpu.0.gpu‚34.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚34.0 gpu.0.powerWatts‚116.069' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.213800000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒË¯W4’!/ _wandbruntime‚11071  _runtime‚11071çÔX¹Ç“!Á _runtime‚11071.066117895 _step‚499 & lr-AdamW/pg1‚0.0003775115151098916 & lr-AdamW/pg2‚0.0003775115151098916 trainer/global_step‚6099 $ _timestamp‚1.788480864310598e+09G[µ/Ì”!Æ & lr-AdamW/pg2‚0.0003775115151098916 trainer/global_step‚6099 $ _timestamp‚1.788480864310598e+09 _runtime‚11071.066117895 _step‚499 & lr-AdamW/pg1‚0.0003775115151098916ó¶„L4•!/ _wandbruntime‚11071  _runtime‚11071ÌϾž(–!P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20471436798464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20471362763264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚900177379783680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚986876747021824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55760387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884808643117104e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚971996132280832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153043205 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 _step‚500 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚986873109969408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14854507961628 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚971996105889280 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14880614740992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137760421 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14877004080128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14877004080128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚963414086229744 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 trainer/global_step‚6099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚986876747021824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82000034 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129498501 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚948563186930800 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129498495 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153043205 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129498495 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚986873109969408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚879705942985216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14880614740992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚723 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚971996132280832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚879705891782144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚948563212358768 _runtime‚11071.067427664 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024Èzï—!’P _step‚500 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚986873109969408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 29392819712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14854507961628 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚971996105889280 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14880614740992 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137760421 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚2090 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14877004080128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14877004080128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚963414086229744 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 trainer/global_step‚6099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚986876747021824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82000034 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129498501 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚948563186930800 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129498495 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153043205 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129498495 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚986873109969408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚879705942985216 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14880614740992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚723 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚971996132280832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚879705891782144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚948563212358768 _runtime‚11071.067427664 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20471436798464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20471362763264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚900177379783680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚986876747021824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55760387 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74035200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884808643117104e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚971996132280832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153043205 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 3610660864 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187ôozö4˜!/ _wandbruntime‚11072  _runtime‚11072ª.`Á'™!—N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282560351 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129518925 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129518925 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚972149883448320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14856914265122 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282586723 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚972149883448320 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20474742088192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚963570455865026 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153067798 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153067798 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282586723 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153041432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14853305602230 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14879414561792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚972149909839872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14883025235968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚948713516171936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282560351 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚963566821774166 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚987032935075840 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137782165 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14883025235968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14879414561792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137782879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚987029298010112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55770035 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚879845233374208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚879845182171136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82012844 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884808654782665e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424¨Í3"'š!œN REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55769326 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123127808 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884808654782665e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 534404096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 4586471424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282560351 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129518925 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚714 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 KBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129518925 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚972149883448320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14856914265122 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282586723 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚972149883448320 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20474742088192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚963570455865026 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153067798 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153067798 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282586723 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153041432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14853305602230 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14879414561792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚972149909839872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14883025235968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚948713516171936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282560351 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚963566821774166 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚987032935075840 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137782165 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14883025235968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14879414561792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137782879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚71924736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚987029298010112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55770035 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚879845233374208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚879845182171136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82012844 P^œ!Ø 0loss/FRt4/assignment_loss‚3.2068736929642525 1loss/FBt2/assignment_loss‚0.21157620751519754 /loss/FRt2/detection_loss‚1.6136565449146123 /loss/FBt1/detection_loss‚3.1569585601488748 .loss/FBt4/detection_loss‚2.319459710802351 epoch‚5 2loss/SRqqt1/assignment_loss‚1.0108625496721875 1loss/SRqqt2/assignment_loss‚2.126933860151391 1loss/FBt1/assignment_loss‚0.28732980504476774 1loss/FBt4/assignment_loss‚0.10913506074125291 1loss/SRqqt2/detection_loss‚2.3164152063821493 _runtime‚ 11072.2358833 0loss/FRt3/assignment_loss‚3.6564325761795042 2loss/SRqqt3/assignment_loss‚1.4918062697723509 1loss/SRqqt4/detection_loss‚2.4652435985895305 trainer/global_step‚6099 1loss/FBt3/assignment_loss‚0.41097691802561337 1loss/SRqqt1/detection_loss‚0.8142096956686701 1loss/SRqqt3/detection_loss‚2.4188697004318236 /loss/FBt2/detection_loss‚0.7930310290459901 /loss/FBt3/detection_loss‚2.5512907645281624 0loss/FRt2/assignment_loss‚2.5153632278625784 1loss/SRqqt4/assignment_loss‚1.308183607716973 /loss/FRt4/detection_loss‚3.5605207744397616 _step‚502 /loss/FRt1/assignment_loss‚1.321976877895056 % _timestamp‚1.7884808654805765e+09 /loss/FRt1/detection_loss‚1.8669167919312264 .loss/FRt3/detection_loss‚3.090043625831604 &loss/total_loss‚44.624066656255884Ïàc!Ý 0loss/FRt4/assignment_loss‚3.2068736929642525 1loss/FBt2/assignment_loss‚0.21157620751519754 /loss/FRt2/detection_loss‚1.6136565449146123 /loss/FBt1/detection_loss‚3.1569585601488748 .loss/FBt4/detection_loss‚2.319459710802351 epoch‚5 2loss/SRqqt1/assignment_loss‚1.0108625496721875 1loss/SRqqt2/assignment_loss‚2.126933860151391 1loss/FBt1/assignment_loss‚0.28732980504476774 1loss/FBt4/assignment_loss‚0.10913506074125291 1loss/SRqqt2/detection_loss‚2.3164152063821493 _runtime‚ 11072.2358833 0loss/FRt3/assignment_loss‚3.6564325761795042 2loss/SRqqt3/assignment_loss‚1.4918062697723509 1loss/SRqqt4/detection_loss‚2.4652435985895305 trainer/global_step‚6099 1loss/FBt3/assignment_loss‚0.41097691802561337 1loss/SRqqt1/detection_loss‚0.8142096956686701 1loss/SRqqt3/detection_loss‚2.4188697004318236 /loss/FBt2/detection_loss‚0.7930310290459901 /loss/FBt3/detection_loss‚2.5512907645281624 0loss/FRt2/assignment_loss‚2.5153632278625784 1loss/SRqqt4/assignment_loss‚1.308183607716973 /loss/FRt4/detection_loss‚3.5605207744397616 _step‚502 /loss/FRt1/assignment_loss‚1.321976877895056 % _timestamp‚1.7884808654805765e+09 /loss/FRt1/detection_loss‚1.8669167919312264 .loss/FRt3/detection_loss‚3.090043625831604 &loss/total_loss‚44.624066656255884öûÖ{[–ž!‚(:‹ îšèÔ—› x' disk.nvme0n1.out‚275242.44189453125 disk.sda2.in‚0 disk.dm-1.out‚ 337.484375 disk.sda.in‚ 0.00390625 disk.drbd1260.out‚1.65625$ memory_percent‚99.93913173675537 network.recv‚ 3.518183e+06 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.00390625 proc.memory.rssMB‚ 3264.9375* proc.memory.percent‚19.927597045898438 cpu‚25.238300080184114) disk./.usagePercent‚78.13743966209627% disk./.usageGB‚1164.3043174743652 disk.sda1.in‚0$ disk.nvme0c0n1.in‚28121.34765625) disk.nvme0c0n1.out‚275242.44189453125 disk.dm-0.in‚28121.34765625$ disk.dm-0.out‚275242.44189453125 disk.sda.out‚ 337.484375" disk.nvme0n1.in‚28121.34765625 disk.dm-1.in‚ 0.00390625 disk.sda3.out‚ 337.484375 disk.drbd1260.in‚0& proc.memory.availableMB‚ 9.97265625 proc.cpu.threads‚17 network.sent‚ 1.7547961e+07“§ÛHŸ!‚(:’ îšèÔñ¡¡x gpu.0.gpu‚52.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.585' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.516999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­¾Ä,• !‚(:Š ýšèÔ‹ÅÐx disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 337.859375# memory_percent‚99.9417781829834! proc.memory.rssMB‚ 3265.359375& disk.nvme0n1.out‚275708.6025390625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 337.859375* proc.memory.percent‚19.930171966552734 network.sent‚ 1.7656668e+07 network.recv‚ 3.524649e+06" disk.nvme0n1.in‚28192.55859375 disk.sda1.out‚0% proc.memory.availableMB‚ 9.5390625 cpu‚25.238269325235127 proc.cpu.threads‚17) disk./.usagePercent‚78.13901334040925 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.66015625$ disk.nvme0c0n1.in‚28192.55859375( disk.nvme0c0n1.out‚275708.6025390625# disk.dm-0.out‚275708.6025390625 disk.sda.in‚ 0.00390625 disk.sda.out‚ 337.859375$ disk./.usageGB‚1164.327766418457 disk.sda2.out‚0 disk.dm-0.in‚28192.55859375£Ù(œ¡!‚(:‘ ýšèÔ‚Úâx gpu.0.gpu‚62.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚249.58' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.916000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ῇ¢!‚(:ü Œ›èÔ¹Ãz disk.drbd1260.in‚0 disk.nvme0n1.in‚ 28300.296875 disk.sda2.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚276151.3564453125 disk.dm-0.in‚ 28300.296875 disk.sda.out‚ 338.4609375) proc.memory.percent‚19.93255615234375) disk./.usagePercent‚78.10988941129901 disk.dm-1.out‚ 338.4609375 disk.drbd1260.out‚ 1.66015625& disk.nvme0n1.out‚276151.3564453125# memory_percent‚99.9420166015625 proc.memory.availableMB‚9.5 cpu‚25.238253840602752 proc.cpu.threads‚17# disk.dm-0.out‚276151.3564453125 network.sent‚ 1.7661648e+07 network.recv‚ 3.527833e+06 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 28300.296875 disk.sda.in‚ 0.00390625 disk.sda3.out‚ 338.4609375 proc.memory.rssMB‚3265.75$ disk./.usageGB‚1163.893798828125 disk.sda2.out‚0 disk.dm-1.in‚ 0.00390625 disk.sda3.in‚ 0.00390625ü}ïÇ’£!‚(:‡ Œ›èÔ펱z gpu.0.gpu‚48.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚273.853' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7706 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ÂG¤!‚(:’ ››èÔþãÿw gpu.0.gpu‚83.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚42.0 gpu.0.powerWatts‚247.172' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.434400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ÃW ¥!‚(: ››èÔãâñx) disk./.usagePercent‚78.03729028272407 disk.dm-0.in‚ 28300.296875 disk.drbd1260.in‚0$ memory_percent‚99.94382858276367$ proc.memory.availableMB‚9.203125* proc.memory.percent‚19.934391975402832 proc.cpu.threads‚17%q?™[ disk./.usageGB‚1162.8120193481445) disk.nvme0c0n1.out‚276658.85693359375 disk.sda.out‚ 338.78515625 disk.sda3.in‚ 0.00390625 disk.sda3.out‚ 338.78515625 disk.drbd1260.out‚ 1.6640625 disk.sda2.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 28300.296875$ disk.dm-0.out‚276658.85693359375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚ 338.78515625 disk.sda.in‚ 0.00390625 disk.nvme0n1.in‚ 28300.296875 disk.sda1.out‚0# proc.memory.rssMB‚ 3266.05078125 network.sent‚ 1.7668058e+07' disk.nvme0n1.out‚276658.85693359375 disk.sda2.out‚0 cpu‚25.238232584085303 network.recv‚ 3.532732e+06Eðò<Ž¦!‚(:ƒ ª›èԘЄx disk.sda2.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 28300.296875 disk.dm-0.in‚ 28300.296875 network.sent‚ 1.7672807e+07 disk.sda2.in‚0$ disk.dm-0.out‚276837.30615234375 disk.dm-1.in‚ 0.00390625 disk.dm-1.out‚339.3125 disk.sda.in‚ 0.00390625 disk.sda.out‚339.3125 disk.sda3.in‚ 0.00390625) disk./.usagePercent‚78.03728516257338 disk.nvme0n1.in‚ 28300.296875& proc.memory.availableMB‚ 9.23046875# proc.memory.rssMB‚ 3266.24609375 cpu‚25.238230116084384 network.recv‚ 3.535361e+06 disk.drbd1260.out‚ 1.6640625 disk.sda1.in‚0 disk.sda3.out‚339.3125# memory_percent‚99.9436616897583) proc.memory.percent‚19.93558406829834 proc.cpu.threads‚17) disk.nvme0c0n1.out‚276837.30615234375% disk./.usageGB‚1162.8119430541992 disk.drbd1260.in‚0' disk.nvme0n1.out‚276837.30615234375é¿…œ§!‚(:‘ ª›èÔö×Òy gpu.0.gpu‚47.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚40.0 gpu.0.powerWatts‚195.537' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚39.107400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0) tX4¨!/ _wandbruntime‚11157  _runtime‚11157§èŸÊ©!Ä ' lr-AdamW/pg1‚0.00037286049394469784 ' lr-AdamW/pg2‚0.00037286049394469784 trainer/global_step‚6149 % _timestamp‚1.7884809503639684e+09 _runtime‚11157.120228432 _step‚503#ÖÚaϪ!É ' lr-AdamW/pg1‚0.00037286049394469784 ' lr-AdamW/pg2‚0.00037286049394469784 trainer/global_step‚6149 % _timestamp‚1.7884809503639684e+09 _runtime‚11157.120228432 _step‚503÷Ý2FÝ4«!/ _wandbruntime‚11157  _runtime‚11157éWc®(¬!P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26340 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32002 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚979684983841280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121043968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20638165837312 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚956079674415568 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994682526418432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚971054511132704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154248567 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 _runtime‚11157.125418054 _step‚504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994686163470848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130519695 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138846869 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚994686163470848 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚979684983841280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14997542577152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 4509787136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130519701 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 28808577024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130519701 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82640289 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚907308698113024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚979685010232832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284794608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚907308577069056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56207298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56206585 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15001153238016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154274907 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154248567 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130519695 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14997542577152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154274907 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20638235678208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚886670462434816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69840896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14974836717136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15001153238016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138847587 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14971228054452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26391552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130519701 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82640289 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚907308698113024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚51203072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚979685010232832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284794608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚907308577069056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 28808577024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 33395048448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 4586471424 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 25702182912 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1532 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 25702182912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56207298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56206585 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 534404096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15001153238016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154274907 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26391552 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154248567 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 3610660864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3921 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130519695 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 4509787136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 3637052416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14997542577152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154274907 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20638235678208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚886670462434816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚69840896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14974836717136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 33395048448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15001153238016 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138847587 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 3380285952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14971228054452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121043968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20638165837312 MDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚2187 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚718 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚877 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚956079674415568 H?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2187 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994682526418432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 3608662684 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚971054511132704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 4507766904 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154248567 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 _runtime‚11157.125418054 _step‚504 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994686163470848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1221 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130519695 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138846869ø7"½4®!/ _wandbruntime‚11159  _runtime‚11159oûjóœ%¯!–N PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138869176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚971207246677334 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 _runtime‚11159.515493394 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130540125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56216082 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130540119 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚979838761400320 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154299500 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154273134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788480952755544e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154273134 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚956230003656704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15003563732992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚956229978228736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284839625 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82653089 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284839625 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886809752823808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20641579246080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14977243020630 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 _step‚505 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚994838714459136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284813253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138868483 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130540119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚994842351524864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚979838787791872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚907451332069888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚907451211039232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20641509418496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 trainer/global_step‚6149 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚994842351524864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130540125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82653094 GADe-(߀viceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14999953058816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154299500¼CÂé!'°!›N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56215394 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284839625 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886809752823808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20641579246080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14977243020630 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 _step‚505 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚994838714459136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284813253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138868483 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 28808577024 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130540119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚994842351524864 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚979838787791872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚907451332069888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚907451211039232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20641509418496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 trainer/global_step‚6149 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚994842351524864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26391552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130540125 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82653094 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14999953058816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154299500 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 138869176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚3719 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚971207246677334 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 _runtime‚11159.515493394 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130540125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56216082 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130540119 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚979838761400320 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154299500 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154273134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788480952755544e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154273134 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚956230003656704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15003563732992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 I@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚2187 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚956229978228736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284839625 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82653089 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 _runtime‚11234.512127501 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56652730 _step‚508 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131540895 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚893634930681344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131540901 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155506609 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155480269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1002491942867456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139933269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15121691735040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131540901 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139932538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131540895 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287047510 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚893634981884416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15121691735040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20804921921536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56652004 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚978691301945012 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155506609 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155480269 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15091556809960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚978687667854360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚963596111044400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32002 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 28808577024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚3719 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 3637052416 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139933269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚3719 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15121691735040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131540901 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139932538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71938048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131540895 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2187 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287047510 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚893634981884416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 33395048448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15121691735040 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 3896742400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20804921921536 ICDÞþ­3D eviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56652004 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚978691301945012 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚3719 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26340 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155506609 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155480269 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3634090652 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15091556809960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚978687667854360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚963596111044400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 4586471424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3921 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 28880505161 _runtime‚11234.512127501 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56652730 _step‚508 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131540895 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚893634930681344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131540901 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155506609 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155480269 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1002491942867456 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 29392819712 JÆg4 'Ç!šN RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚914582467919872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20808320871424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚963746465713504 trainer/global_step‚6199 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 _runtime‚11235.531215761 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15120491555840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15093963113246 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131561319 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚724 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚987527639352320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚963746440285536 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139954165 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978844037489642 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155504836 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚914582593144832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287066155 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚509 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287092527 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚893774221070336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83293339 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15124102230016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚987527665743872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1002648130908160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83293344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1002651767973888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978840403398782 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15097571776138 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155504836 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0£ók´%'È!ŸN UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 25702182912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1221 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 3896742400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 28880505161 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 33395048448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 4586471424 _runtime‚11235.531215761 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15120491555840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 3634090860 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 25168546816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15093963113246 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚3719 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131561319 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 29392819712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 3610674176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚724 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚987527639352320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125224960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚963746440285536 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139954165 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978844037489642 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26366 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 3637065728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3921 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 28808577024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155504836 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 3610674176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 4509787136 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚2090 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚914582593144832 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 3380285952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚3719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26391552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 25702182912 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚51203072 F=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚2187 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 3637065728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287066155 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚509 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287092527 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚877 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32002 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚893774221070336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚74021888 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 3608662892 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 4509787136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83293339 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1532 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15124102230016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚719 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚987527665743872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 4507766904 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1002648130908160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83293344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1002651767973888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978840403398782 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15097571776138 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26366 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155504836 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚3719 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 33395048448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 29392819712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 33395048448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 28808577024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 4586471424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚914582467919872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20808320871424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚963746465713504 trainer/global_step‚6199ý“Í°´4É!/ _wandbruntime‚11235  _runtime‚11235öá›]Ê!× /loss/FRt4/assignment_loss‚3.233778721519879 2loss/SRqqt4/assignment_loss‚1.1385513207521931 /loss/FBt2/detection_loss‚0.8342584363584008 .loss/FRt2/detection_loss‚1.802737149908099 1loss/SRqqt2/detection_loss‚3.0957606931527457 0loss/FRt1/assignment_loss‚1.5911223320290446 .loss/FRt1/detection_loss‚1.694868471130576 0loss/SRqqt3/detection_loss‚2.297209980090459 .loss/FBt4/detection_loss‚2.455651874725635 epoch‚5 0loss/FRt2/assignment_loss‚2.2549852396166603 1loss/SRqqt3/assignment_loss‚1.908132686962684 1loss/SRqqt1/detection_loss‚0.9852412439098484 /loss/FBt1/detection_loss‚1.9183380842208861 /loss/FBt3/detection_loss‚3.0885180168681674 %loss/total_loss‚43.91997122689745 0loss/FBt4/assignment_loss‚0.2227201511268504 2loss/SRqqt2/assignment_loss‚1.8194283582270145 /loss/FRt3/detection_loss‚3.2474773033805517 /loss/FRt4/detection_loss‚3.0836783881698335 trainer/global_step‚6199 _runtime‚11235.534983051 _step‚510 2loss/SRqqt1/assignment_loss‚0.9608332754129119 1loss/FBt2/assignment_loss‚0.07089048307922616 1loss/FBt3/assignment_loss‚0.16757903464248455 0loss/FRt3/assignment_loss‚3.5251524448394775 1loss/FBt1/assignment_loss‚0.12072974129296199 1loss/SRqqt4/detection_loss‚2.4023277954808595 % _timestamp‚1.7884810287763638e+09—£ÆÐbË!Ü /loss/FRt3/detection_loss‚3.2474773033805517 /loss/FRt4/detection_loss‚3.0836783881698335 trainer/global_step‚6199 _runtime‚11235.534983051 _step‚510 2loss/SRqqt1/assignment_loss‚0.9608332754129119 1loss/FBt2/assignment_loss‚0.07089048307922616 1loss/FBt3/assignment_loss‚0.16757903464248455 0loss/FRt3/assignment_loss‚3.5251524448394775 1loss/FBt1/assignment_loss‚0.12072974129296199 1loss/SRqqt4/detection_loss‚2.4023277954808595 % _timestamp‚1.7884810287763638e+09 /loss/FRt4/assignment_loss‚3.233778721519879 2loss/SRqqt4/assignment_loss‚1.1385513207521931 /loss/FBt2/detection_loss‚0.8342584363584008 .loss/FRt2/detection_loss‚1.802737149908099 1loss/SRqqt2/detection_loss‚3.0957606931527457 0loss/FRt1/assignment_loss‚1.5911223320290446 .loss/FRt1/detection_loss‚1.694868471130576 0loss/SRqqt3/detection_loss‚2.297209980090459 .loss/FBt4/detection_loss‚2.455651874725635 epoch‚5 0loss/FRt2/assignment_loss‚2.2549852396166603 1loss/SRqqt3/assignment_loss‚1.908132686962684 1loss/SRqqt1/detection_loss‚0.9852412439098484 /loss/FBt1/detection_loss‚1.9183380842208861 /loss/FBt3/detection_loss‚3.0885180168681674 %loss/total_loss‚43.91997122689745 0loss/FBt4/assignment_loss‚0.2227201511268504 2loss/SRqqt2/assignment_loss‚1.8194283582270145þgPÑpÝÌ!‚(:’ “œèÔÉâ¤z gpu.0.gpu‚56.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚39.52156066894531* gpu.0.memoryAllocatedBytes‚ 33948762112 gpu.0.temp‚48.0 gpu.0.powerWatts‚287.816' gpu.0.enforc